Skip to content

Time series ensemble models #1105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 14 commits into from
Mar 17, 2021
Merged

Time series ensemble models #1105

merged 14 commits into from
Mar 17, 2021

Conversation

TremaMiguel
Copy link
Contributor

Related Issuse or bug

  • Info about Issue or bug

Fixes: [#1094]

Describe the changes you've made

Major changes include:

  • Custom Implementation of sktime EnsembleForecaster with mean, median and voting methods. link

Minor changes include:

  • Forces Series conversion when calling sktime metrics. Pycaret Experiment handles data as DataFrame internally. link

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Code style update (formatting, local variables)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

The following tests are included

  • test_blend_models test parametrically the TimeSeriesExperiment.blend_models() method. link

with the addition of the following fixtures to share components through tests:

  • load_setup fixture to share an experiment to test TimeSeriesExperiment different functionalities. link
  • load_ts_models fixture to share timeseries models to test TimeSeriesExperiment different functionalities link

Describe if there is any unusual behaviour of your code(Write NA if there isn't)

blend_model fails with sktime PolyTrend Model, when retrieving the id of the model container through a dictionary, None is returned as the key.

Checklist:

  • My code follows the style guidelines of this project.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have made corresponding changes to the documentation.
  • My changes generate no new warnings.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • Any dependent changes have been merged and published in downstream modules.

@TremaMiguel TremaMiguel requested review from Yard1 and ngupta23 March 10, 2021 15:21
@TremaMiguel TremaMiguel self-assigned this Mar 10, 2021
ngupta23
ngupta23 reviewed Mar 11, 2021
View reviewed changes
Copy link
Collaborator

@ngupta23 ngupta23 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @TremaMiguel , In the notebook, it would be god to show an example of the 3 Ensemble methds that show difference in results. Currently all 3 show the same result.

Also, is this expected? Shouldn't the one with weights give a different result?

pycaret/tests/test_time_series.py
)
assert np.all(y_pred.index == expected)


@pytest.mark.parametrize("method", _ENSEMBLE_METHODS)
def test_blend_model(load_setup, load_models, method):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a unit test for blending of only a subset of models?

ts_blender = exp.blend_models([arima_model, naive_model], optimize='MAPE_ts')

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ngupta23 could you clarify some details about what are we expecting to test by pairs?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in the example notebook you have an example with blending only a subset of models. It would be good to check this through a unit test to make sure that it does not always produce the same results as when all models are blended.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, @ngupta23 I added a functional test to assert that predictions from mean, median and voting ensemble differ considering all the models available in the ts module.

Yard1
Yard1 approved these changes Mar 11, 2021
View reviewed changes
Copy link
Member

@Yard1 Yard1 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I've got nothing to add that @ngupta23 hasn't said already. Just please be careful with formatting changes unrelated to actual PR.

@TremaMiguel
Copy link
Contributor Author

Hi @TremaMiguel , In the notebook, it would be god to show an example of the 3 Ensemble methds that show difference in results. Currently all 3 show the same result.

Also, is this expected? Shouldn't the one with weights give a different result?

@ngupta23 sorry, the voting ensemble called the median method by mistake, change it to appropiate voting method and results differ from mean and median.

@ngupta23 ngupta23 merged commit fb453d3 into time_series Mar 17, 2021
@ngupta23 ngupta23 added time_series Topics related to the time series and removed time_series_dev labels Sep 19, 2021
@ngupta23 ngupta23 deleted the time_series_ensemble_models branch September 25, 2021 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Yard1 Yard1 Yard1 approved these changes

@ngupta23 ngupta23 ngupta23 approved these changes

Assignees

@TremaMiguel TremaMiguel

Labels
time_series Topics related to the time series
Projects
None yet
Milestone
Time Series Alpha Release
Development

Successfully merging this pull request may close these issues.
3 participants
@TremaMiguel @Yard1 @ngupta23