[common] Log a warning if mlflow autologging is enabled #4925
Conversation
| except Exception: | ||
| # gracefully handle cases where autologging_is_disabled is not available | ||
| pass | ||
| except Exception: |
There was a problem hiding this comment.
This could, in theory, be narrowed down to an ImportWarning, but I don't want to accidentally break the entire AG training process if import mlflow fails with some other spectacular exception.
| @@ -193,3 +193,32 @@ def reset_logger_for_remote_call(verbosity: int): | |||
| # Limiting the verbosity of the BaggedEnsembleModel logger to 10 to avoid | |||
| # duplicated messages about fitting. | |||
| set_logger_verbosity(verbosity=min(verbosity, 1), logger=bem_logger) | |||
|
|
|||
|
|
|||
| def warn_if_mlflow_autologging_is_enabled(logger: Optional[logging.Logger] = None): | |||
There was a problem hiding this comment.
Any ideas for how/whether we can test this method in our unittests without installing mlflow?
There was a problem hiding this comment.
rather test the good path?
(pseudocode)
def test():
try:
import mlflow
pytest.fail("mlflow not expected to be installed in test env")
except:
assert warn_if_mlflow_autologging_is_enabled() does not log
|
Job PR-4925-ef0b265 is done. |
| @@ -1043,6 +1043,7 @@ def fit( | |||
|
|
|||
| verbosity = kwargs.get("verbosity", self.verbosity) | |||
| set_logger_verbosity(verbosity) | |||
| warn_if_mlflow_autologging_is_enabled(logger=logger) | |||
There was a problem hiding this comment.
is it enough to call this in fit? why not somewhere in an init file?
There was a problem hiding this comment.
The patched metrics only cause problems when saving objects with pickle (which is only required during training). Calling methods like leaderboard or evaluate should work normally.
| @@ -193,3 +193,32 @@ def reset_logger_for_remote_call(verbosity: int): | |||
| # Limiting the verbosity of the BaggedEnsembleModel logger to 10 to avoid | |||
| # duplicated messages about fitting. | |||
| set_logger_verbosity(verbosity=min(verbosity, 1), logger=bem_logger) | |||
|
|
|||
|
|
|||
| def warn_if_mlflow_autologging_is_enabled(logger: Optional[logging.Logger] = None): | |||
There was a problem hiding this comment.
rather test the good path?
(pseudocode)
def test():
try:
import mlflow
pytest.fail("mlflow not expected to be installed in test env")
except:
assert warn_if_mlflow_autologging_is_enabled() does not log
ef0b265 to
06e0679
Compare
|
Job PR-4925-06e0679 is done. |
Issue #, if available: Fixes #4914
Description of changes:
autologging is enabled on platforms like databricks, so users often encounter this issue with AG.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.