Description

I'm working with the Time-series module in AutoGluon v0.8.2 and v1.0.0, aiming to assess performance sensitivity to random initialization by feeding alternative seeds during model training. Despite attempts, changing the random_seed or model-specific seed parameter doesn't affect the outcomes (identical results) for models like DeepAR, TemporalSpatialTransformer, etc.

The documentation suggests that random_seed should ensure reproducible results across most models, except for non-deterministic GPU operations. However, my attempts to modify the seed as per the documentation have not yielded different results.

random_seed : int, optional
If provided, fixes the seed of the random number generator for all models. This guarantees reproducible
results for most models (except those trained on GPU because of the non-determinism of GPU operations).

Thanks for your help!

Attempts to resolve

I've tried setting the seed both directly in the fit method and within hyperparameters, as shown below, without any change in the outcomes:

      predictor.fit(
          train_data,
          random_seed=random_seed,
          hyperparameters={
              model_type: model_params,
          },
      )

model_params = {"seed":random_seed)
      predictor.fit(
          train_data,
          hyperparameters={
              model_type: model_params,
          },
      )

Note: model_type and model_params are placeholders for the specific models and parameters used.

Expected vs. Actual Behavior

I expected that changing the random_seed or seed would result in variations in model training outcomes, reflecting sensitivity to initialization. However, the results remain unchanged regardless of the seed value. I did however observe differences in forecasts from v0.8.2 to v1.0.0.

Environment

AutoGluon version: v0.8.2, v1.0.0
Python version: 3.7
Operating System: Linux

References

I've reviewed similar issues (e.g., #425, #1454) without finding a solution that addresses my problem.

References

I've reviewed similar issues (e.g., #425, #1454) without finding a solution that addresses my problem.