This is an issue which isn't very obvious. Sometimes after running get_offline_features, then running materialize_features API, the materialize_features API will not be successful, and in many cases there's no values in the online store such as Redis.

This only happens when using databricks.

This is caused by the fact that if the databricks configuration and it's not a string (i.e. end users use a dict to provide all the required configurations), there's a line:

submission_params = self.config_template

Since self.config_template is a dict, this is actually a reference rather than a copy of self.config_template. In the code later, submission_params will be modified and the value will be persisted across jobs, which will cause unexpected behaviors.