Skip to content

Automatically handle add_metric with string target labels #3491

@bottyBotz

Description

@bottyBotz

pycaret version checks

Issue Description

Similar issue to those identified in #3351.

Specificity is just the recall for the negative class in a binary classification task. Therefore we just return scikit-learn's recall function where the pos_label is 0. We utilize the zero_division parameter to return 1 if there is a zero_division error. It is worth noting that the default behavior of scikit-learn's recall function sets zero_division = 'warn' which is equivalent to zero_division = 0.

Reproducible Example

from pycaret.datasets import get_data
from sklearn.metrics import recall_score
data = get_data('juice')

from pycaret.classification import *
s = setup(data, target = 'Purchase', session_id = 123)

# create a custom function
def specificity(y_true, y_pred):
    return recall_score(y_true
                        , y_pred
                        , pos_label=0
                        , zero_division=1)


# add metric to PyCaret
add_metric('specificity', 'specificity', specificity, greater_is_better = True)

lr = create_model('lr')

Expected Behavior

Expected behavior would be that specificity is correctly calculated.

Actual Results

Actual results are that all zeros are returned.

![image](https://user-images.githubusercontent.com/67231425/233394122-b4954d33-64e8-4d29-901f-a182a9f9c22a.png)

Installed Versions

3.0.0

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions