Fair CRPS #1661
-
|
Many thanks for adding the empirical CRPS to MET in MET #1450. Do you have any plans to implement the 'fair' version of this? I'm increasingly seeing it used in practice. To quote Ferro (2014) "Fair scores for ensemble forecasts", "the original CRPS thus favours ensembles that are sampled from overconfident (i.e. underdispersed) distributions", which is clearly undesirable. There are several equivalent formulas for the fair CRPS (and indeed the original empirical CRPS) in the literature. Perhaps the simplest is equation 4 in Fricker, Ferro, & Stephenson, D.B. (2013), but they all involve the mean absolute difference (aka Gini's mean difference) of the ensemble members. A naive implementation of that involves a double loop over the ensemble members but a better algorithm involves sorting followed by a single loop – see the two formulae near the bottom of the first page of David (1968). This is available for R in the FairCrps function of the SpecsVerification package. There the algorithm is coded in C++ (enscrps.cpp), but it's licensed under the GPL so you probably couldn't reuse the code directly. There are also 'fair' version of the Brier score and the RPS for ensemble forecasts, but they can be calculated from the counts in the existing PCT output line type from Point-Stat and Grid-Stat (I'm just finishing coding this in our VerPy system, which can ingest MET output). Thanks, |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 1 reply
-
|
Hi Roger, And thank you for your suggestion. I've looked over the numerous resources you've provided us (thank you for those!) and tend to agree: a method to account for biasing in current statistical MET output is a worthwhile endeavor. There are a few concerns to address if an issue is created for this and work is performed, though. One of these concerns is the method for calculating fair CRPS. I'm very new to the subject bias-corrected statistical scores, but I am aware that similar to the standard scores (and skill scores), bias-corrected scores can have multiple approaches that account for the various sources of bias. To that concern, does the method proposed in the Fricker, Ferro, & Stephenson, D.B paper present one of the more widely-accepted methods? And does it address the greater concerns of ensemble bias when using CRPS? I'm going to ask around about this to see what we can find on our side, but your input is greatly appreciated. You also noted the requirement for a mean absolute difference in the ensemble members. While not of high concern, I do wonder if a larger (100+) member ensemble would see any significant drop in MET tool performance/run time. |
Beta Was this translation helpful? Give feedback.
-
|
Hi John, To clarify, all the methods for calculating the fair CRPS give the same results and all include a term containing the mean absolute difference of the ensemble values. Given that you've already coded the empirical CRPS it seems simplest to use the formula in the Fricker, Ferro & Stephenson paper (given elsewhere too) that uses this as a starting point and subtracts an adjustment. So-called 'fair scores' address one particular form of bias in verification metrics for ensemble forecasts, namely that from assessing raw NWP output from a finite-size ensemble. This is an entirely separate issue from other sources of bias in the CRPS, which I believe mainly concern observation error and representativeness. I think that's a more difficult issue to tackle in general, as it requires knowledge of the properties of the observations such as error distributions. However I'm not really in a position to judge the relative importance of the two. I don't believe you'd see any significant drop in MET performance/run time. As the difference between the fair and conventional scores decreases with ensemble size, the fair CRPS is less likely to be useful for very large ensembles in any case. But I see your implementation of the conventional ensemble CRPS in compute_crps_emp() starts by sorting the ensemble values, and sorting is the only part of an efficient algorithm for the mean absolute difference that isn't O(N) in the ensemble size. The calculation could be done straightforwardly with an extra line or two in the existing loop over ensemble members: from equation (4) of Ferro, Fricker & Stephenson and the equation before last on the first page of David (1968) provided the ensemble values Perhaps I should emphasise that I'm not suggesting this fair CRPS should replace the conventional (empirical) CRPS, as the two are applicable in different situations - for example the fair CRPS is not appropriate for assessing final products or post-processed forecasts, only raw NWP ensemble output. |
Beta Was this translation helpful? Give feedback.
-
|
Roger, I was able to go into today and add a new Github issue describing this. It really is a worthwhile statistic to include and as you've illustrated, shouldn't take too much additional effort, given that we already have CRPS in the ECNT line type. |
Beta Was this translation helpful? Give feedback.
Roger,
I was able to go into today and add a new Github issue describing this. It really is a worthwhile statistic to include and as you've illustrated, shouldn't take too much additional effort, given that we already have CRPS in the ECNT line type.
If the issue is written correctly and addresses the concerns you've brought up in this Discussions, feel free to mark this as Answered and I'll close out the Discussions. You can then track the Github issue for continued progress.