Skip to content

[tabular] Add plot_leaderboard.py #4907

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 21, 2025
Merged

[tabular] Add plot_leaderboard.py #4907

merged 1 commit into from
Feb 21, 2025

Conversation

Innixma
Copy link
Contributor

@Innixma Innixma commented Feb 19, 2025 •
edited
Loading

Issue #, if available:

Description of changes:

  • Add experimental plot_leaderboard.py, with functions to visualize AutoGluon's performance over time for a given task.

Code

from autogluon.tabular import TabularPredictor
from autogluon.tabular.experimental.plot_leaderboard import plot_leaderboard_with_predictor

data_root = 'https://autogluon.s3.amazonaws.com/datasets/Inc/'
predictor_example = TabularPredictor(label="class").fit(train_data=data_root + "train.csv", time_limit=60)
figure, lb = plot_leaderboard_from_predictor(predictor=predictor_example, test_data=data_root + "test.csv", ensemble=True)
with pd.option_context("display.max_rows", None, "display.max_columns", None, "display.width", 1000):
    print(lb)
figure.savefig("example_leaderboard_plot.png")

Output

Figure

plot_leaderboard_example

Leaderboard

                         model  metric_error_test  metric_error_val eval_metric  pred_time_test  pred_time_val   fit_time  pred_time_test_marginal  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer    best_model_so_far  best_idx_so_far  time_so_far  metric_error_val_so_far  metric_error_test_so_far  metric_error_val_so_far_ens  metric_error_test_so_far_ens  best_idx_so_far_ens best_model_so_far_ens
fit_order                                                                                                                                                                                                                                                                                                                                                                                                                        
1               KNeighborsUnif           0.228375            0.2264    accuracy        0.025831       0.006755   1.991074                 0.025831                0.006755           1.991074            1       True       KNeighborsUnif                1     1.991074                   0.2264                  0.228375                       0.2264                      0.228375                    1        KNeighborsUnif
2               KNeighborsDist           0.239124            0.2348    accuracy        0.020363       0.009389   0.373825                 0.020363                0.009389           0.373825            1       True       KNeighborsUnif                1     2.364899                   0.2264                  0.228375                       0.2220                      0.231242                    2        KNeighborsDist
3                   LightGBMXT           0.128570            0.1208    accuracy        0.009888       0.004432   2.298011                 0.009888                0.004432           2.298011            1       True           LightGBMXT                3     4.662910                   0.1208                  0.128570                       0.1188                      0.128365                    3            LightGBMXT
4                     LightGBM           0.126523            0.1176    accuracy        0.007721       0.003136   1.387891                 0.007721                0.003136           1.387891            1       True             LightGBM                4     6.050801                   0.1176                  0.126523                       0.1148                      0.125397                    4              LightGBM
5             RandomForestGini           0.140649            0.1388    accuracy        0.353945       0.083391   0.944486                 0.353945                0.083391           0.944486            1       True             LightGBM                4     6.995287                   0.1176                  0.126523                       0.1148                      0.125397                    4              LightGBM
6             RandomForestEntr           0.142389            0.1416    accuracy        0.262797       0.084268   1.041147                 0.262797                0.084268           1.041147            1       True             LightGBM                4     8.036434                   0.1176                  0.126523                       0.1148                      0.125397                    4              LightGBM
7                     CatBoost           0.124475            0.1176    accuracy        0.038252       0.012311   7.587049                 0.038252                0.012311           7.587049            1       True             LightGBM                4    15.623483                   0.1176                  0.126523                       0.1140                      0.123861                    7              CatBoost
8               ExtraTreesGini           0.146586            0.1472    accuracy        0.328208       0.230981   1.363113                 0.328208                0.230981           1.363113            1       True             LightGBM                4    16.986596                   0.1176                  0.126523                       0.1140                      0.123861                    7              CatBoost
9               ExtraTreesEntr           0.149350            0.1476    accuracy        0.378278       0.084387   1.429653                 0.378278                0.084387           1.429653            1       True             LightGBM                4    18.416250                   0.1176                  0.126523                       0.1140                      0.123861                    7              CatBoost
10             NeuralNetFastAI           0.140444            0.1356    accuracy        0.159357       0.030853  29.459874                 0.159357                0.030853          29.459874            1       True             LightGBM                4    47.876123                   0.1176                  0.126523                       0.1140                      0.123861                    7              CatBoost
11                     XGBoost           0.122940            0.1180    accuracy        0.024146       0.009660   2.800007                 0.024146                0.009660           2.800007            1       True             LightGBM                4    50.676130                   0.1176                  0.126523                       0.1140                      0.123861                    7              CatBoost
12              NeuralNetTorch           0.147507            0.1484    accuracy        0.040015       0.016440   6.818561                 0.040015                0.016440           6.818561            1       True             LightGBM                4    57.494691                   0.1176                  0.126523                       0.1140                      0.123861                    7              CatBoost
13         WeightedEnsemble_L2           0.124680            0.1140    accuracy        0.984978       0.392656  50.811939                 0.002897                0.000726           0.093962            2       True  WeightedEnsemble_L2               13    57.588653                   0.1140                  0.124680                       0.1140                      0.123861                    7              CatBoost

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@Innixma Innixma added this to the 1.3 Release milestone Feb 19, 2025
@github-actions GitHub Actions
Copy link

Job PR-4907-91f1280 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-4907/91f1280/index.html

shchur
shchur approved these changes Feb 19, 2025
View reviewed changes
tabular/src/autogluon/tabular/experimental/plot_leaderboard.py
Comment on lines +41 to +47
Parameters
----------
leaderboard: pd.DataFrame

Returns
-------
leaderboard_stats: pd.DataFrame
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: These are already documented by the type hints, so the docstring could be removed.

@Innixma Innixma merged commit 199cae8 into autogluon:master Feb 21, 2025
27 checks passed
@Innixma Innixma deleted the plot_leaderboard branch April 16, 2025 21:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@shchur shchur shchur approved these changes

@prateekdesai04 prateekdesai04 prateekdesai04 approved these changes

Assignees
No one assigned
Labels
module: tabular
Projects
Tabular 1.3
Status: Done
Milestone
1.3 Release
Development

Successfully merging this pull request may close these issues.
3 participants
@Innixma @shchur @prateekdesai04