Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Job PR-4054-3c7fe15 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-4054/3c7fe15/index.html

Thank you so much for moving so quickly to adopt my suggested changes!

Just want to follow up on the reason for requiring historical period in test_data: let's say the prediction_length of my predictor is 30 days, why can't I pass a test_data which also has 30 days of data to .leaderboard() since that is the only data I want to calculate the evaluation score on? Thanks!

I have the same question: why is it required that data must be at least prediction_length + 1? Why not prediction_length instead of prediction_length+1?

Thank you so much for moving so quickly to adopt my suggested changes!

Just want to follow up on the reason for requiring historical period in test_data: let's say the prediction_length of my predictor is 30 days, why can't I pass a test_data which also has 30 days of data to .leaderboard() since that is the only data I want to calculate the evaluation score on? Thanks!

@lleiou @shashankumar2812 When we call predictor.evaluate(data), essentially, the following happens:

1. We split data into two disjoint parts:
   • future_data (last prediction_length time steps of each time series)
   • past_data (everything before the last prediction_length steps of each time series).
2. We call forecast = predictor.predict(past_data)
3. We compare forecast & future data score = eval_metric(future_data, forecast).

If we only provide the last prediction_length observations of each time series to predictor.evaluate, then after the split our past_data will be empty, so the predictor won't be able to generate a forecast for the forecast horizon.

We opted for this more general design (where both past data + future data is required when calling evaluate) since it allows us to evaluate the predictor on different time series / different train-test splits. If we went for the alternative approach, where predictor.evaluate accepts just the future data, this would mean that we can only evaluate the predictor on the exact same time series used for training, using only the future values starting from the end of the train data.

Thanks @shchur for your answer