Fix AUC metric #3935

celestinoxp · 2024-03-04T17:08:41Z

some details have not been updated to support the latest scikit-learn 1.4 code

confirm/update code
fix errors
make sure tests are testing metrics correctly (needs create more tests?)

celestinoxp · 2024-03-05T10:11:48Z

@Yard1 @moezali1 @tvdboom @glemaitre @ogrisel @thomasjpfan @lorentzenchr @adrinjalali

something is wrong with AUC metrics... i have no idea how to fix this pull-request...

ogrisel · 2024-03-05T11:12:56Z

Could you please provide a minimal reproducer on synthetic data that ideally only involves scikit-learn? Working on crafting such a reproducer will likely help you understand what's going on.

celestinoxp · 2024-03-06T08:41:50Z

Could you please provide a minimal reproducer on synthetic data that ideally only involves scikit-learn? Working on crafting such a reproducer will likely help you understand what's going on.

from pycaret.datasets import get_data
juice = get_data('juice')
from pycaret.classification import *
exp_name = setup(data = juice,  target = 'Purchase')
best_model = compare_models()

celestinoxp · 2024-03-11T22:01:00Z

@ngupta23 can you help?

thomasjpfan · 2024-03-12T17:59:26Z

pycaret/containers/metrics/classification.py

@@ -115,10 +116,11 @@ def __init__(
            if scorer
            else pycaret.internal.metrics.make_scorer_with_error_score(
                score_func,
-                needs_proba=target == "pred_proba",
-                needs_threshold=target == "threshold",
+                response_method=None,


If this is calling scikit-learn's make_scorer under the covers, then you can pass in the response_method directly here.

if target == "pred" response_method = "predict" elif target == "pred_proba": response_method = "predict_proba" else: # threshold response_method = "decision_function" ... else pycaret.internal.metrics.make_scorer_with_error_score( score_func, response_method=response_method, greater_is_better=greater_is_better, error_score=0.0, )

I tested but still not working...
logs.log show:

2024-03-12 18:16:58,428:WARNING:C:\Users\celes\anaconda3\lib\site-packages\pycaret\internal\metrics.py:196: FitFailedWarning: Metric 'make_scorer(roc_auc_score, response_method=('decision_function', 'predict_proba'), average=weighted, multi_class=ovr)' failed and error score 0.0 has been returned instead. If this is a custom metric, this usually means that the error is in the metric code. Full exception below: Traceback (most recent call last): File "C:\Users\celes\anaconda3\lib\site-packages\pycaret\internal\metrics.py", line 188, in _score return super()._score( File "C:\Users\celes\anaconda3\lib\site-packages\sklearn\metrics\_scorer.py", line 345, in _score y_pred = method_caller( File "C:\Users\celes\anaconda3\lib\site-packages\sklearn\metrics\_scorer.py", line 87, in _cached_call result, _ = _get_response_values( File "C:\Users\celes\anaconda3\lib\site-packages\sklearn\utils\_response.py", line 210, in _get_response_values y_pred = prediction_method(X) File "C:\Users\celes\anaconda3\lib\site-packages\pycaret\internal\pipeline.py", line 341, in predict_proba Xt = transform.transform(Xt) File "C:\Users\celes\anaconda3\lib\site-packages\sklearn\utils\_set_output.py", line 295, in wrapped data_to_wrap = f(self, X, *args, **kwargs) File "C:\Users\celes\anaconda3\lib\site-packages\pycaret\internal\preprocess\transformers.py", line 233, in transform X = to_df(X, index=getattr(y, "index", None)) File "C:\Users\celes\anaconda3\lib\site-packages\pycaret\utils\generic.py", line 103, in to_df data = pd.DataFrame(data, index, columns) File "C:\Users\celes\anaconda3\lib\site-packages\pandas\core\frame.py", line 822, in __init__ mgr = ndarray_to_mgr( File "C:\Users\celes\anaconda3\lib\site-packages\pandas\core\internals\construction.py", line 319, in ndarray_to_mgr values = _prep_ndarraylike(values, copy=copy_on_sanitize) File "C:\Users\celes\anaconda3\lib\site-packages\pandas\core\internals\construction.py", line 575, in _prep_ndarraylike values = np.array([convert(v) for v in values]) ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part. warnings.warn( 2024-03-12 18:16:58,428:WARNING:C:\Users\celes\anaconda3\lib\site-packages\sklearn\metrics\_classification.py:1561: UserWarning: Note that pos_label (set to 'MM') is ignored when average != 'binary' (got 'weighted'). You may use labels=[pos_label] to specify a single positive class.

@thomasjpfan Can you help to investigate if the problem is with pycaret or scikit-learn? I'm doing tests on my laptop but I'm not sure where the error is to fix...

I do not have the bandwidth to investigate.

I do not have the bandwidth to investigate.

but can you talk to someone on the scikit-learn side for support?

You need to debug to see if it is a pycaret bug or a scikit-learn bug. If it is a scikit-learn bug, then open an issue with a minimal reproduce that only involves scikit-learn.

#3935 (comment) is not a valid reproducer for scikit-learn because it is still using pycaret.

celestinoxp · 2024-03-13T18:37:24Z

Does any good heart have the time and ability to fix this problem?

Ping: @Yard1 @ngupta23 @tvdboom @moezali1 @TremaMiguel @daikikatsuragawa @timho102003 @andrinbuerli @goodwanghan @drmario-gh @AJarman @reza1615 @batmanscode @sherpan @IncubatorShokuhou @wkuopt @cspartalis @ryanxjhan @jinensetpal

celestinoxp · 2024-03-20T10:58:39Z

@Aloqeely can you give help fix bugs in pycaret?

Aloqeely · 2024-03-21T21:01:37Z

Sorry, I am not familiar with PyCaret
Good luck!

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

CMobley7 · 2024-08-01T21:16:34Z

@Yard1 , This problem appears to still exist for multiclass classification. If you use the simple example below, 7 of the 16 models will return 0.0000 for AUC. These models include lr, qda, lda, gbc, ada, ridge, and svm. Also, custom_metric, like the one below, works for binary classification if you add **kwargs, as I suggested in #3973, but give 0.0000 for multiclass classification. I think this may be related to the same issue.

Simple Example

from pycaret.datasets import get_data
from pycaret.classification import ClassificationExperiment

data = get_data('iris')
exp = ClassificationExperiment()
exp.setup(data, target = 'species', session_id = 123)
exp.compare_models()

Custom Metric

from pycaret.datasets import get_data
from pycaret.classification import ClassificationExperiment
from sklearn.metrics import fbeta_score

def f2_score(y_true, y_pred, **kwargs):
    """
    Calculate the F2 score.

    Args:
        y_true (1d array-like): The true labels.
        y_pred (1d array-like): The predicted labels.
        **kwargs: Additional arguments for fbeta_score.

    Returns:
        float: The F2 score.
    """
    return fbeta_score(y_true, y_pred, beta=2, **kwargs)

data = get_data('iris')
exp = ClassificationExperiment()
exp.setup(data, target = 'species', session_id = 123)
exp.add_metric(id="f2", name="F2", score_func=f2_score, target="pred", average="macro")
exp.compare_models()

I also tried leaving out average="macro" from the add_metric and updating f2_score to check if y_true had more than 2 unique values. If it did, the var average="macro" along with **kwargs was sent into fbeta_score. This didn't fix the 0.0000 issue either.

celestinoxp · 2024-08-03T20:21:23Z

@CMobley7 can you do a PR fixing this?

paolodep36 · 2025-05-13T15:50:48Z

Still this problem, very annoying

celestinoxp and others added 2 commits March 4, 2024 17:05

Update classification.py

4acf3a4

Merge branch 'master' into fix_auc_metrics

e7bcd0a

houpdelta mentioned this pull request Mar 4, 2024

All result AUC = 0 with compare_model #3932

Closed

3 tasks

fix classification.py

6e43b23

celestinoxp mentioned this pull request Mar 7, 2024

Support roc_auc_score() for multi-class without probability estimates scikit-learn/scikit-learn#18676

Open

thomasjpfan reviewed Mar 12, 2024

View reviewed changes

small fix

ff5536d

moezali1 requested a review from Yard1 April 25, 2024 19:47

Yard1 added 3 commits April 27, 2024 21:03

Merge branch 'master' into fix_auc_metrics

81f0cd8

Fix AUC metric (predict_proba)

86c5918

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

Lint

a6de9d0

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>

Yard1 changed the title ~~[WIP] fix auc metric~~ Fix AUC metric Apr 28, 2024

Update pipeline.py

cb59dc2

Yard1 merged commit 9ee0cf4 into pycaret:master Apr 28, 2024

Fix AUC metric #3935

Fix AUC metric #3935

Uh oh!

Conversation

celestinoxp commented Mar 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

celestinoxp commented Mar 5, 2024

Uh oh!

ogrisel commented Mar 5, 2024

Uh oh!

celestinoxp commented Mar 6, 2024

Uh oh!

celestinoxp commented Mar 11, 2024

Uh oh!

thomasjpfan Mar 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

celestinoxp Mar 12, 2024

Choose a reason for hiding this comment

Uh oh!

celestinoxp Mar 13, 2024

Choose a reason for hiding this comment

Uh oh!

thomasjpfan Mar 13, 2024

Choose a reason for hiding this comment

Uh oh!

celestinoxp Mar 13, 2024

Choose a reason for hiding this comment

Uh oh!

thomasjpfan Mar 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

celestinoxp commented Mar 13, 2024

Uh oh!

celestinoxp commented Mar 20, 2024

Uh oh!

Aloqeely commented Mar 21, 2024

Uh oh!

CMobley7 commented Aug 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Simple Example

Custom Metric

Uh oh!

celestinoxp commented Aug 3, 2024

Uh oh!

paolodep36 commented May 13, 2025

Uh oh!

Uh oh!

celestinoxp commented Mar 4, 2024 •

edited

Loading

thomasjpfan Mar 12, 2024 •

edited

Loading

thomasjpfan Mar 13, 2024 •

edited

Loading

CMobley7 commented Aug 1, 2024 •

edited

Loading