Global Precision/Recall/F1 Callback #433

jchen42703 · 2019-10-10T19:37:52Z

Description

PrecisionRecallF1ScoreMeter: tracks TP, FP, and FN for each loader, and calculates precision (ppv), recall (tpr), and f1-score, based on those metrics.
PrecisionRecallF1ScoreCallback: callback for PrecisionRecallF1ScoreMeter. Modeled after the AUCCallback & AccuracyCallback (multiple metrics).
Example logs for a 4 class multi-label classifier

Related Issue

N/A

Type of Change

Examples / docs / tutorials / contributors update
Bug fix (non-breaking change which fixes an issue)
Improvement (non-breaking change which improves an existing feature)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist

I have read the CODE_OF_CONDUCT document.
I have read the CONTRIBUTING document.
I have checked the code-style using make check-style.
I have written the docstring in Google format for all the method and classes that I used.
I have checked the docs using make check-docs.

Keeps track of global true positives, false positives, and false negatives for each epoch and calculates precision, recall, and F1-score based on those metrics. Currently, for binary cases only (use multiple instances for multi-label).

Calculates the global precision (positive predictive value or ppv), recall (true positive rate or tpr), and F1-score per class for each loader. Currently, supports binary and multi-label cases.

…global-metrics Updating with the latest changes to remove possibility of git diff bugs

For PrecisionRecallF1ScoreCallback

…uments

TezRomacH · 2019-10-10T19:55:05Z

Hi! Thank you for your PR!
Can you please run the make codestyle command to convert the code to the style used in Catalyst and make our CI tests pass?

jchen42703 · 2019-10-10T20:37:55Z

Yep, I'll give it a go tonight!

jchen42703 · 2019-10-10T22:33:34Z

Quick self-reminder:
Double check that the average values are computed correctly (prec_recall_f1score[prefix] = metric_ v. prec_recall_f1score[prefix].append(metric_))

Edit: Fixed

Passes `make codestyle`.

…ending the metric This caused metrics to be averaged incorrectly because we were just calling np.mean(float) instead of np.mean(list of floats)

catalyst/dl/meters/ppv_tpr_f1_meter.py

Scitator · 2019-10-11T08:04:18Z

catalyst/dl/callbacks/metrics/ppv_tpr_f1.py

+    def on_batch_end(self, state: RunnerState):
+        logits: torch.Tensor = state.output[self.output_key].detach().float()
+        targets: torch.Tensor = state.input[self.input_key].detach().float()
+        probabilities: torch.Tensor = torch.sigmoid(logits)


I think, we could parametrise the used function during callback initialization

I might be misinterpreting what you're saying, but are you asking to make a reusable function to do the ops above and initialize it as an attribute during callback initialization?

I mean, you can parametrize activation function like here

I think it would be more consistent, but I don't think PrecisionRecallF1ScoreCallback should inherit MetricCallback/MultiMetricCallback because PrecisionRecallF1ScoreCallback calculates the metrics for the entire loader instead of for each batch (maybe make a separate base global callback?). I'm curious on your thoughts?
(I'll add the option to specify the activation function but without the super.init(...) in the meantime)

I think we can create new abstraction like "DatasetMetricCallback", that need need to collect statistic during on_batch_end and also some additional calculations on_loader_end. We have the same case with AUCCallback, so... looks like we need something general :)

#450 <- Related PR

I'll try to get to it tomorrow!

Do we need to add .cpu()? Otherwise it throws error:
TypeError: can't convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

Scitator · 2019-10-11T08:05:33Z

If you have any questions – do not hesitate to ask :)

…ils.get_activation_fn

…ds for PrecisionRecallF1ScoreMeter Did so for increased clarity.

…oreMeter Still need to create tests for the callback, but this is a rough start.

Scitator · 2019-10-13T17:06:36Z

@jchen42703 PR looks good, the only thing – check the codestyle please.

jchen42703 · 2019-10-13T18:15:57Z

@Scitator Should I add in a test for the callback (not just the meter + metrics) as well?

Scitator · 2019-10-14T09:59:52Z

catalyst/dl/callbacks/metrics/ppv_tpr_f1.py

+    def on_batch_end(self, state: RunnerState):
+        logits: torch.Tensor = state.output[self.output_key].detach().float()
+        targets: torch.Tensor = state.input[self.input_key].detach().float()
+        activation_fn = get_activation_fn(self.activation)


I think you can move it to init

A callback that tracks metrics through meters and prints metrics for each class on `state.on_loader_end`. **Have not tested yet**

jchen42703 · 2019-10-16T21:46:02Z

Gonna try to adapt PrecisionRecallF1ScoreCallback and AUCCallback to MeterMetricsCallback tomorrow and test to see if it works properly

Should pass

Cleaner, reduces repeat code.

Cleaner, reduces repeat code

jchen42703 · 2019-10-17T23:51:58Z

They work for binary (num_classes=2) and multi-label cases, but not when num_classes=1, so I'm considering dropping that portion of MeterMetricsCallback.
Also, I'm considering refactoring confusionmeter.py using #450's catalyst.utils.confusion_matrix.py to track all of the stats so that we can expand this class to #450's callbacks. I could be overcomplicating this though.

Scitator · 2019-10-18T11:14:03Z

@jchen42703 You are right, currently we do not support num_classes==1, so I think it's good idea to add assert num_classes > 1 for now.

Meanwhile, MeterMetricsCallback looks really good 👍

jchen42703 · 2019-10-20T23:31:41Z

Gonna try to do some last minute cleanup:

making num_classes an attribute for MeterMetricsCallback instead of repeating for AUCCallback and PrecisionRecallF1ScoreCallback
Also, adding the class_names attribute for MeterMetricsCallback <- big woops for excluding this on my end
testing on multi-class cases (num_classes > 3) <- should work bc num_classes=2 works properly
- logs for:
Codestyle fixes

Edit:

I think the confusionmatrix.py refactoring deserves a separate PR unless this PR gets merged w/ Add dice jaccard callbacks #450's

Removed unnecessary imports + overindented

Did so because catalyst does not currently support self.num_classes == 1 and len(probabilities.shape) == 1; tensors have no channels and you get a bunch of CUDA and memory pinning errors.

…back` Did so because doing so makes it clear to anyone implementing a child of `MeterMetricsCallback` that you need to specify `class_names` and `num_classes` for the callback to work properly. Also added a check for num_classes == 1 (should be > 1).

…zation changes (class_names and num_classes) Did so for PrecisionRecallF1ScoreCallback and AUCCallback

Scitator · 2019-10-29T19:50:54Z

Awesome PR!

jchen42703 added 5 commits October 9, 2019 21:23

Created PrecisionRecallF1ScoreMeter

7b36190

Keeps track of global true positives, false positives, and false negatives for each epoch and calculates precision, recall, and F1-score based on those metrics. Currently, for binary cases only (use multiple instances for multi-label).

Created PrecisionRecallF1ScoreCallback

845f647

Calculates the global precision (positive predictive value or ppv), recall (true positive rate or tpr), and F1-score per class for each loader. Currently, supports binary and multi-label cases.

Merge branch 'master' of https://github.com/jchen42703/catalyst into …

86aa7b3

…global-metrics Updating with the latest changes to remove possibility of git diff bugs

Update __init__.py

5f7d579

For PrecisionRecallF1ScoreCallback

Added more documentation for the PrecisionRecallF1ScoreCallback's arg…

98e3d56

…uments

TezRomacH added enhancement New feature or request Hacktoberfest labels Oct 10, 2019

jchen42703 added 2 commits October 10, 2019 18:43

Fixing up codestyle for PEP 8

021038b

Passes `make codestyle`.

Fixed bug where the list of metrics was set as a float instead of app…

b55ce9d

…ending the metric This caused metrics to be averaged incorrectly because we were just calling np.mean(float) instead of np.mean(list of floats)

Scitator reviewed Oct 11, 2019

View reviewed changes

jchen42703 added 4 commits October 12, 2019 00:45

Moved f1score, precision, and recall to the top

e378de3

Added the option to specify the activation function using catalyst.ut…

1d14fd1

…ils.get_activation_fn

Added more documentation for the .add(), .reset(), and .value() metho…

341f381

…ds for PrecisionRecallF1ScoreMeter Did so for increased clarity.

Created tests for precision, recall, f1score, and PrecisionRecallF1Sc…

6c1437d

…oreMeter Still need to create tests for the callback, but this is a rough start.

Fixed up codestyle

11e21c7

Scitator reviewed Oct 14, 2019

View reviewed changes

Created MeterMetricsCallback

429c39c

A callback that tracks metrics through meters and prints metrics for each class on `state.on_loader_end`. **Have not tested yet**

jchen42703 added 3 commits October 17, 2019 19:27

Added in missing imports, self.class_names, and spaces for codestyle

043e131

Should pass

Adapted PrecisionRecallF1ScoreCallback to inherit MeterMetricsCallback

36a16f1

Cleaner, reduces repeat code.

Adapted AUCCallback to inherit MeterMetricsCallback

43b4bbc

Cleaner, reduces repeat code

jchen42703 added 5 commits October 20, 2019 19:38

Added MeterMetricsCallback for the .callback imports

15b2929

Fixed codestyle

9553556

Removed unnecessary imports + overindented

Removed the section for when num_classes==1

82e8faa

Did so because catalyst does not currently support self.num_classes == 1 and len(probabilities.shape) == 1; tensors have no channels and you get a bunch of CUDA and memory pinning errors.

Adapted the callbacks to work with the MeterMetricCallback's initiali…

1cb5a7c

…zation changes (class_names and num_classes) Did so for PrecisionRecallF1ScoreCallback and AUCCallback

jchen42703 requested a review from Scitator October 23, 2019 02:37

Scitator approved these changes Oct 29, 2019

View reviewed changes

Scitator merged commit c6ea0fc into catalyst-team:master Oct 29, 2019

smivv mentioned this pull request Oct 30, 2019

Triplet loss epic #459

Closed

Uh oh!

Global Precision/Recall/F1 Callback #433

Global Precision/Recall/F1 Callback #433

Uh oh!

Conversation

jchen42703 commented Oct 10, 2019 • edited by TezRomacH Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue

Type of Change

Checklist

Uh oh!

TezRomacH commented Oct 10, 2019

Uh oh!

jchen42703 commented Oct 10, 2019

Uh oh!

jchen42703 commented Oct 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Scitator Oct 11, 2019

Choose a reason for hiding this comment

Uh oh!

jchen42703 Oct 11, 2019

Choose a reason for hiding this comment

Uh oh!

Scitator Oct 11, 2019

Choose a reason for hiding this comment

Uh oh!

jchen42703 Oct 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Scitator Oct 14, 2019

Choose a reason for hiding this comment

Uh oh!

jchen42703 Oct 14, 2019

Choose a reason for hiding this comment

Uh oh!

jchen42703 Oct 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

smivv Oct 30, 2019

Choose a reason for hiding this comment

Uh oh!

Scitator commented Oct 11, 2019

Uh oh!

Scitator commented Oct 13, 2019

Uh oh!

jchen42703 commented Oct 13, 2019

Uh oh!

Scitator Oct 14, 2019

Choose a reason for hiding this comment

Uh oh!

jchen42703 commented Oct 16, 2019

Uh oh!

jchen42703 commented Oct 17, 2019

Uh oh!

Scitator commented Oct 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jchen42703 commented Oct 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Scitator commented Oct 29, 2019

Uh oh!

Uh oh!

jchen42703 commented Oct 10, 2019 •

edited by TezRomacH

Loading

jchen42703 commented Oct 10, 2019 •

edited

Loading

jchen42703 Oct 12, 2019 •

edited

Loading

jchen42703 Oct 16, 2019 •

edited

Loading

Scitator commented Oct 18, 2019 •

edited

Loading

jchen42703 commented Oct 20, 2019 •

edited

Loading