is [1, 1, 1, 1] more intuitive than [0.25, 0.25, 0.25, 0.25]? When one says weight my mind immediately jumps to something that sums to 1. This is especially apparent if we need lead time: [0, 0, 2, 2] vs [0, 0, .5, .5].

This is mostly about efficiency and numerical stability rather than being intuitive. We always have a matrix errors of shape [num_items, prediction_length]. There are two ways to apply different weights per item / per time step.

Current approach (no weights = all weights are equal to 1)

If we have an array horizon_weight of shape [1, prediction_length] where values sum up to prediction_length, and an array item_weight of shape [num_items, 1], where values sum up to num_items, the averaging logic is quite simple

errors: np.ndarray,  # shape [num_items, prediction_length]
if horizon_weight is not None:
    errors *= horizon_weight
if item_weight is not None:
    errors *= item_weight
return np.nanmean(errors)

Alternative approach (no weights = all weights are equal to 1/prediction_length or 1/num_items)

If instead horizon_weight and item_weight each summed up to 1, the code would be a lot less elegant. We won't be able to just optionally multiple the errors with some weights, but rather we will need to perform reduction after each multiplication. We won't be able to use np.average for this reduction since it doesn't support NaN values, so we will need to us np.nansum. So the code will look something like

errors: np.ndarray,  # shape [num_items, prediction_length]
if horizon_weight is not None:
    errors = np.nansum(errors * horizon_weight, axis=1)
else:
    errors = np.nanmean(errors, axis=1)
# now `errors` has shape [num_items]
if item_weight is not None:
    return np.nansum(errors * item_weight)
else:
    return np.nanmean(errors)

This looks less elegant to me.

Also, if the weights are uniform, I suspect that np.nansum(errors * horizon_weight, axis=1) is less numerically stable than np.nanmean(errors, axis=1) (especially if prediction_length is very large).

Sorry for being late to the party. I understand it's similar in spirit to how sample_weight works in most stats software. LGTM.

Do you mean that sample_weight usually adds up to 1, or that it usually adds up to len(y)? In sklearn it seems to be latter (https://scikit-learn.org/stable/modules/generated/sklearn.utils.class_weight.compute_sample_weight.html).

I meant the latter.

One breaking change is that the prediction_length must be set as an attribute of the TimeSeriesScorer.

This, for example, means that changes are required to the code in the tutorials

mse = MeanSquaredError()
mse_score = mse(
  data=test_data,
  predictions=predictions,
  prediction_length=predictor.prediction_length,
  target=predictor.target,
)

to

mse = MeanSquaredError(prediction_length=predictor.prediction_length)
mse_score = mse(
  data=test_data,
  predictions=predictions,
  target=predictor.target,
)

We could still allow passing prediction_length via kwargs here for backward compatibility (so both options above would work). Do you think this makes sense?

def __call__(..., **kwargs):
    prediction_length = kwargs.get("prediction_length", self.prediction_length)

would be good to have backward compatibility although I am not sure how many users work directly with metrics.

I've added a backward-compatible option to pass the prediction_length here.

I don't see why an AbstractTimeSeriesModel should be able to score with different metrics (other than its own eval_metric), so I removed this functionality. We only use it in unit tests, in normal usage scoring with custom metrics is handled by TimeSeriesTrainer.

makes sense. in the future: why should AbstractTimeSeriesModel even be aware of its metric :) ?

would be good to have backward compatibility although I am not sure how many users work directly with metrics.