[Tabular] Introduce compile_models function to tabular predictor #2260

liangfu · 2022-10-26T21:03:12Z

Description of changes:

Following #2225, this PR takes a further step to introduce compile_models function to TabularPredictor. This helps

Decouple compile_models from fit and save function from tabular predictor.
Remove compiler option from hyperparameters, since hyperparameters are primarily designed for model fitting. Compiler options are more suitable for post-fit process.
Add support to compile WeightedEnsembleModel.

Also, it is required to persist model before model compilation. Therefore, the convention for model compilation would be

predictor = TabularPredictor(label=label, path=save_path).fit(train_data)

# Compile and persist the best model for efficient prediction without loading model in every predict() call.
# NOTE: leave `with_ancestors` argument to be the default value True, in case the best model is an ensemble model.
compiler_configs = {RFModel: {'compiler': 'onnx'}}
perdictor.compile_models(models='best', compiler_configs=compiler_configs)
predictor.persist_models(models='best')

# Perform prediction just as before
y_pred = predictor.predict(test_data)

To compare performance with/without zipmap

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

github-actions · 2022-10-26T23:42:50Z

Job PR-2260-eeb875b is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2260/eeb875b/index.html

Innixma

Added initial review

core/src/autogluon/core/trainer/abstract_trainer.py

core/src/autogluon/core/models/ensemble/weighted_ensemble_model.py

tabular/src/autogluon/tabular/predictor/predictor.py

tabular/src/autogluon/tabular/models/rf/compilers/onnx.py

tabular/tests/conftest.py

core/src/autogluon/core/models/abstract/abstract_model.py

github-actions · 2022-11-01T20:39:11Z

Job PR-2260-57ae20f is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2260/57ae20f/index.html

github-actions · 2022-11-02T07:05:27Z

Job PR-2260-710e6ba is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2260/710e6ba/index.html

core/src/autogluon/core/trainer/abstract_trainer.py

Innixma · 2022-11-02T19:25:16Z

core/src/autogluon/core/models/abstract/abstract_model.py

+        if type(self) in compiler_configs:
+            configs = compiler_configs[type(self)]
+        elif self.name in compiler_configs:
+            configs = compiler_configs[self.name]


Why are we allowing passing of other model compiler configs? This should be handled upstream, and compiler_configs should simply be {"compiler": "onnx"} in this context.

This is kind of a model-specific compiler configration. For ensemble models, we could use a specific backend for a specific model. For instance, an ensemable model with RF and TORCH_NN, I think we could configure the compiler to optimize RF with onnx, while optimize TORCH_NN model with tvm, in order to maximize potential performance benefits. Of course, we can still use the top-level compiler config if model-specific compiler is not specified.

a model-specific compiler configration

This shouldn't be available at the AbstractModel level, we only need the configuration for the model itself in this context, not others.

Model-specific config logic has been moved outside of abstract_model.

Innixma · 2022-11-02T19:27:47Z

core/src/autogluon/core/models/abstract/abstract_model.py

+        if type(self) in compiler_configs:
+            configs = compiler_configs[type(self)]
+        elif self.name in compiler_configs:
+            configs = compiler_configs[self.name]


core/src/autogluon/core/models/abstract/abstract_model.py

core/src/autogluon/core/models/ensemble/stacker_ensemble_model.py

tabular/src/autogluon/tabular/models/rf/compilers/onnx.py

tonyhoo · 2022-11-03T22:41:32Z

core/src/autogluon/core/models/abstract/abstract_model.py

-        if path is None:
-            path = self.path
-        file_path = path + self.model_file_name
+    def can_compile(self, compiler_configs=None):


Do we need compiler_config to be passed by the caller each time?

For now, I think the answer is yes. It makes the interface a little bit difficult to configure, since users may not aware the underlying compilers that are supported.

In the long term, I think we may introduce the presets argument to the compile_models method.

To support edge devices, we could support presets like

rpi

jetson-tx1

For cloud deployment, presets could be something like

aws-lambda # for online prediction, batch_size=1

sagemaker-endpoint # for batch prediction, batch_size=1024

github-actions · 2022-11-03T23:48:10Z

Job PR-2260-c454f37 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2260/c454f37/index.html

(cherry picked from commit eeb875b)

…comments

…rface

github-actions · 2022-11-04T08:25:07Z

Job PR-2260-42781d4 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2260/42781d4/index.html

github-actions · 2022-11-04T08:25:22Z

Job PR-2260-9c7e698 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2260/9c7e698/index.html

github-actions · 2022-11-05T08:31:35Z

Job PR-2260-75b68b7 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-2260/75b68b7/index.html

Innixma

Looks great! Thanks for the contribution!

Innixma · 2022-11-04T19:40:41Z

tabular/src/autogluon/tabular/predictor/predictor.py

+
+        Parameters
+        ----------
+        compiler_configs : dict, default = {}


Missing description of what the key is

Innixma · 2022-11-04T19:41:32Z

tabular/src/autogluon/tabular/predictor/predictor.py

+        Compile models for accelerated prediction.
+        This can be helpful to reduce prediction latency and improve throughput.
+
+        Note that this is currently an experimental feature, the supported alternative compiler can be ['native', 'onnx'].


Explain that this can take significant time to compile.

Innixma · 2022-11-04T19:42:10Z

tabular/src/autogluon/tabular/predictor/predictor.py

+        Parameters
+        ----------


Missing models parameter

Innixma · 2022-11-04T19:42:23Z

tabular/src/autogluon/tabular/predictor/predictor.py

+        Parameters
+        ----------


Missing with_ancestors parameter

Innixma · 2022-11-04T19:43:50Z

tabular/src/autogluon/tabular/predictor/predictor.py

+        Compile models for accelerated prediction.
+        This can be helpful to reduce prediction latency and improve throughput.
+
+        Note that this is currently an experimental feature, the supported alternative compiler can be ['native', 'onnx'].


first line of docstring should be [Experimental]

Innixma · 2022-11-04T19:49:24Z

core/src/autogluon/core/trainer/abstract_trainer.py

@@ -1447,6 +1497,7 @@ def _add_model(self, model: AbstractModel, stack_name: str = 'core', level: int
        self.model_graph.add_node(
            model.name,
            fit_time=model.fit_time,
+            compile_time=model.compile_time,


Add # FIXME: This won't accurately reflect compile_time if added prior to compiling occurring. Need to update this value after the model is compiled.

Innixma · 2022-11-04T21:34:28Z

tabular/src/autogluon/tabular/predictor/predictor.py

@@ -1950,6 +1950,29 @@ def feature_importance(self, data=None, model=None, features=None, feature_stage
            fi_df[low_str] = pd.Series(ci_low_dict)
        return fi_df

+    def compile_models(self, models='best', with_ancestors=True, compiler_configs=None):


Why is compiler_configs default None if it crashes?

predictor.compile_models()

Traceback (most recent call last): File "/Users/neerick/workspace/virtual/autogluon38/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3433, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "<ipython-input-2-05108f5d756e>", line 1, in <module> runfile('/Users/neerick/workspace/code/autogluon-scratch/scripts/run_adult_compile.py', wdir='/Users/neerick/workspace/code/autogluon-scratch/scripts') File "/Users/neerick/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-1/223.7255.83/PyCharm 2022.3 EAP.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 198, in runfile pydev_imports.execfile(filename, global_vars, local_vars) # execute the script File "/Users/neerick/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-1/223.7255.83/PyCharm 2022.3 EAP.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "/Users/neerick/workspace/code/autogluon-scratch/scripts/run_adult_compile.py", line 37, in <module> predictor.compile_models() File "/Users/neerick/workspace/code/autogluon/tabular/src/autogluon/tabular/predictor/predictor.py", line 1974, in compile_models self._trainer.compile_models(model_names=models, with_ancestors=with_ancestors, compiler_configs=compiler_configs) File "/Users/neerick/workspace/code/autogluon/core/src/autogluon/core/trainer/abstract_trainer.py", line 1166, in compile_models if type(model) in compiler_configs: TypeError: argument of type 'NoneType' is not iterable

Innixma · 2022-11-04T21:44:50Z

tabular/src/autogluon/tabular/predictor/predictor.py

@@ -1950,6 +1950,29 @@ def feature_importance(self, data=None, model=None, features=None, feature_stage
            fi_df[low_str] = pd.Series(ci_low_dict)
        return fi_df

+    def compile_models(self, models='best', with_ancestors=True, compiler_configs=None):


The following code fails. Need to re-load the model after compiling if it was already persisted.

if __name__ == '__main__': from autogluon.tabular import TabularPredictor, TabularDataset path_prefix = 'https://autogluon.s3.amazonaws.com/datasets/AdultIncomeBinaryClassification/' path_train = path_prefix + 'train_data.csv' path_test = path_prefix + 'test_data.csv' label = 'class' sample = 1000 # Number of rows to use to train / infer train_data = TabularDataset(path_train) if sample is not None and (sample < len(train_data)): train_data = train_data.sample(n=sample, random_state=0).reset_index(drop=True) test_data = TabularDataset(path_test) fit_kwargs = dict( train_data=train_data, hyperparameters={ 'RF': {}, }, ) predictor = TabularPredictor( label=label, eval_metric='roc_auc', verbosity=2, ) predictor.fit(**fit_kwargs) predictor.persist_models() leaderboard = predictor.leaderboard(test_data) compiler_configs = {'RandomForest': {'compiler': 'onnx'}} predictor.compile_models(compiler_configs=compiler_configs) predictor.persist_models() leaderboard2 = predictor.leaderboard(test_data)

Because the old model is still persisted, it tries to use the old model but self.model is None so it crashes with:

TabularPredictor saved. To load, use: predictor = TabularPredictor.load("AutogluonModels/ag-20221104_213857/") Persisting 2 models in memory. Models will require 0.09% of memory. model score_test score_val pred_time_test pred_time_val fit_time pred_time_test_marginal pred_time_val_marginal fit_time_marginal stack_level can_infer fit_order 0 RandomForest 0.889409 0.871562 0.128715 0.088706 0.858871 0.128715 0.088706 0.858871 1 True 1 1 WeightedEnsemble_L2 0.889409 0.871562 0.130412 0.091054 0.866322 0.001697 0.002348 0.007451 2 True 2 /Users/neerick/workspace/virtual/autogluon38/lib/python3.8/site-packages/sklearn/utils/deprecation.py:103: FutureWarning: The attribute `n_features_` is deprecated in 1.0 and will be removed in 1.2. Use `n_features_in_` instead. warnings.warn(msg, category=FutureWarning) /Users/neerick/workspace/virtual/autogluon38/lib/python3.8/site-packages/sklearn/utils/deprecation.py:103: FutureWarning: Attribute `n_features_` was deprecated in version 1.0 and will be removed in 1.2. Use `n_features_in_` instead. warnings.warn(msg, category=FutureWarning) The following 2 models were already persisted and will be ignored in the model loading process: ['WeightedEnsemble_L2', 'RandomForest'] No valid unpersisted models were specified to be persisted, so no change in model persistence was performed. Traceback (most recent call last): File "/Users/neerick/workspace/virtual/autogluon38/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3433, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "<ipython-input-2-05108f5d756e>", line 1, in <module> runfile('/Users/neerick/workspace/code/autogluon-scratch/scripts/run_adult_compile.py', wdir='/Users/neerick/workspace/code/autogluon-scratch/scripts') File "/Users/neerick/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-1/223.7255.83/PyCharm 2022.3 EAP.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 198, in runfile pydev_imports.execfile(filename, global_vars, local_vars) # execute the script File "/Users/neerick/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-1/223.7255.83/PyCharm 2022.3 EAP.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "/Users/neerick/workspace/code/autogluon-scratch/scripts/run_adult_compile.py", line 41, in <module> leaderboard2 = predictor.leaderboard(test_data) File "/Users/neerick/workspace/code/autogluon/tabular/src/autogluon/tabular/predictor/predictor.py", line 1570, in leaderboard return self._learner.leaderboard(X=data, extra_info=extra_info, extra_metrics=extra_metrics, File "/Users/neerick/workspace/code/autogluon/tabular/src/autogluon/tabular/learner/abstract_learner.py", line 624, in leaderboard leaderboard = self.score_debug(X=X, y=y, extra_info=extra_info, extra_metrics=extra_metrics, silent=True) File "/Users/neerick/workspace/code/autogluon/tabular/src/autogluon/tabular/learner/abstract_learner.py", line 306, in score_debug model_pred_proba_dict, pred_time_test_marginal = trainer.get_model_pred_proba_dict(X=X, models=all_trained_models_can_infer, record_pred_time=True) File "/Users/neerick/workspace/code/autogluon/core/src/autogluon/core/trainer/abstract_trainer.py", line 811, in get_model_pred_proba_dict model_pred_proba_dict[model_name] = model.predict_proba(X) File "/Users/neerick/workspace/code/autogluon/core/src/autogluon/core/models/abstract/abstract_model.py", line 687, in predict_proba y_pred_proba = self._predict_proba(X=X, **kwargs) File "/Users/neerick/workspace/code/autogluon/tabular/src/autogluon/tabular/models/rf/rf_model.py", line 244, in _predict_proba y_pred_proba = self.model.predict_proba(X) AttributeError: 'NoneType' object has no attribute 'predict_proba'

A workaround (shouldn't be used directly) is to unpersist and persist after the compile_models call:

predictor.compile_models(compiler_configs=compiler_configs) predictor.unpersist_models() predictor.persist_models()

Should instead be done at the trainer level and on a per-model basis.

Innixma · 2022-11-07T21:40:39Z

Note ignore my latest review comments, they are from a draft review prior to me refactoring the code.

Innixma reviewed Oct 26, 2022

View reviewed changes

liangfu force-pushed the compile-1 branch 2 times, most recently from ee8b789 to 5f86f88 Compare October 31, 2022 21:29

liangfu marked this pull request as ready for review October 31, 2022 21:33

liangfu changed the title ~~[WIP][Tabular] Introduce compile_models function to tabular predictor~~ [Tabular] Introduce compile_models function to tabular predictor Oct 31, 2022

liangfu commented Oct 31, 2022

View reviewed changes

core/src/autogluon/core/models/abstract/abstract_model.py Outdated Show resolved Hide resolved

liangfu force-pushed the compile-1 branch from c2f801e to 57ae20f Compare November 1, 2022 18:00

Innixma reviewed Nov 2, 2022

View reviewed changes

core/src/autogluon/core/trainer/abstract_trainer.py Outdated Show resolved Hide resolved

Innixma requested changes Nov 2, 2022

View reviewed changes

liangfu force-pushed the compile-1 branch from 710e6ba to c454f37 Compare November 3, 2022 21:04

tonyhoo reviewed Nov 3, 2022

View reviewed changes

liangfu added 12 commits November 3, 2022 22:36

introduce compile_model to tabular predictor

b83aeb3

(cherry picked from commit eeb875b)

separate compile method from fit and save, and addressed some review …

300232e

…comments

fix compile function

39e7fef

build onnx model without zipmap

469c37b

improve robustness

93fdd4b

address review comments and put compile_time into part of model_info

0a32348

replace _input_types_post_process with _features

eec6cd1

fix compile_time in leaderboard

dd3b152

bug fix in model_info

e626212

add comments

5929215

compile base model via getting its ancestors from compile_models inte…

fe5543c

…rface

move model-specific logic outside of abstract_model

9c7e698

liangfu force-pushed the compile-1 branch from c454f37 to 9c7e698 Compare November 4, 2022 05:36

move model-specific logic outside of abstract_model

42781d4

Innixma mentioned this pull request Nov 5, 2022

Refactored compiler API and logic liangfu/autogluon#1

Merged

Refactored compiler API and logic (#1)

75b68b7

Innixma approved these changes Nov 7, 2022

View reviewed changes

Innixma merged commit a82c09b into autogluon:master Nov 7, 2022

[Tabular] Introduce compile_models function to tabular predictor #2260

[Tabular] Introduce compile_models function to tabular predictor #2260

Uh oh!

Conversation

liangfu commented Oct 26, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 26, 2022

Uh oh!

Innixma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 1, 2022

Uh oh!

github-actions bot commented Nov 2, 2022

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liangfu Nov 4, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 3, 2022

Uh oh!

github-actions bot commented Nov 4, 2022

Uh oh!

github-actions bot commented Nov 4, 2022

Uh oh!

github-actions bot commented Nov 5, 2022

Uh oh!

Innixma left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Innixma commented Nov 7, 2022

Uh oh!

liangfu commented Oct 26, 2022 •

edited

Loading

liangfu Nov 4, 2022 •

edited

Loading