Skip to content

[tabular] Fix eval bug with regression metrics in LightGBM #4174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
May 7, 2024
Merged

[tabular] Fix eval bug with regression metrics in LightGBM #4174

merged 2 commits into from
May 7, 2024

Conversation

Innixma
Copy link
Contributor

@Innixma Innixma commented May 7, 2024

Issue #, if available:
Bug identified in AutoML Grand Prix competition with "r2" metric: https://www.kaggle.com/competitions/playground-series-s4e5/discussion/500783

Description of changes:

  • Fixes a bug that causes LightGBM to internally calculate the metric score incorrectly in regression for custom metrics and metrics other than "mean_absolute_error", "mean_squared_error", and "root_mean_squared_error".
  • For example, "r2" metric will cause LightGBM to perform very badly due to incorrect score calculations.
  • This bug has been present since the origin of the library in 2019.

The logic previously treated the metric as a classification metric such as "accuracy", and would do:

y_hat = np.round(y_hat)

which is incorrect in regression.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@Innixma Innixma added bug Something isn't working module: tabular labels May 7, 2024
@Innixma Innixma added this to the 1.1.1 Release milestone May 7, 2024
@yinweisu
Copy link
Contributor

yinweisu commented May 7, 2024

Previous CI Run Current CI Run

@github-actions GitHub Actions
Copy link

github-actions bot commented May 7, 2024

Job PR-4174-6718456 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-4174/6718456/index.html

prateekdesai04
prateekdesai04 approved these changes May 7, 2024
View reviewed changes
Copy link
Contributor

@prateekdesai04 prateekdesai04 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mglowacki100
Copy link
Contributor

@Innixma It'd be interesting to re-examine results for R^2 mercedes-benz-greener-manufacturing in https://arxiv.org/pdf/2003.06505

@Innixma Innixma merged commit 964e114 into autogluon:master May 7, 2024
@Innixma
Copy link
Contributor Author

Innixma commented May 7, 2024

@Innixma It'd be interesting to re-examine results for R^2 mercedes-benz-greener-manufacturing in https://arxiv.org/pdf/2003.06505

Good eye @mglowacki100! This dataset is part of our new benchmarking suite we are working on, so we might be able to see a bump in performance from this. We have actually got various improvements already on this dataset due to dynamic stacking and other things.

2020 Paper: Top 85%
AutoGluon 1.0: Top 32%
Post This PR: ??%

LennartPurucker pushed a commit to LennartPurucker/autogluon that referenced this pull request Jun 1, 2024
@Innixma @LennartPurucker
[tabular] Fix eval bug with regression metrics in LightGBM (autogluon…
e62631b 
…#4174)
@Innixma Innixma deleted the fix_custom_regression_metrics_lgbm branch April 16, 2025 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@prateekdesai04 prateekdesai04 prateekdesai04 approved these changes

@rey-allan rey-allan Awaiting requested review from rey-allan

Assignees
No one assigned
Labels
bug Something isn't working module: tabular
Projects
None yet
Milestone
1.1.1 Release
Development

Successfully merging this pull request may close these issues.
4 participants
@Innixma @yinweisu @mglowacki100 @prateekdesai04