Skip to content

[BUG]: tuning 'ransac' model does not work #3894

@pascal456

Description

@pascal456

pycaret version checks

Issue Description

  • the 'loss' parameter of RANSACRegressor is not set correctly
  • for example absolute_loss is set when optimizing against "R2"/Default, and squared_loss when optimizing against "RMSE"

Reproducible Example

from pycaret.datasets import get_data
from pycaret.regression import RegressionExperiment

data = get_data("insurance")

sort_metric = "RMSE"

exp = RegressionExperiment()
exp.setup(
    data=data,
    target="charges",
    transform_target=True,
    transformation=True,
)
ransac = exp.create_model("ransac")

and then tuning with

tuned = exp.tune_model(
    estimator=ransac,
    choose_better=True,
    optimize="RMSE",
)

produces sklearn.utils._param_validation.InvalidParameterError: The 'loss' parameter of RANSACRegressor must be a str among {'absolute_error', 'squared_error'} or a callable. Got 'absolute_loss' instead.

or

tuned = exp.tune_model(
    estimator=ransac,
    choose_better=True,
)

produces sklearn.utils._param_validation.InvalidParameterError: The 'loss' parameter of RANSACRegressor must be a str among {'absolute_error', 'squared_error'} or a callable. Got 'squared_loss' instead.

Expected Behavior

tuning with 'ransac' estimator works irrespective of quality of the model; just working

Actual Results

{
	"name": "ValueError",
	"message": "
All the 100 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
26 fits failed with the following error:
Traceback (most recent call last):
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_validation.py\", line 686, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 275, in fit
    fitted_estimator = self._memory_fit(
                       ^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/joblib/memory.py\", line 353, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 68, in _fit_one
    transformer.fit(*args, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/linear_model/_ransac.py\", line 330, in fit
    self._validate_params()
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/base.py\", line 600, in _validate_params
    validate_parameter_constraints(
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/utils/_param_validation.py\", line 97, in validate_parameter_constraints
    raise InvalidParameterError(
sklearn.utils._param_validation.InvalidParameterError: The 'loss' parameter of RANSACRegressor must be a str among {'absolute_error', 'squared_error'} or a callable. Got 'absolute_loss' instead.

--------------------------------------------------------------------------------
24 fits failed with the following error:
Traceback (most recent call last):
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_validation.py\", line 686, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 275, in fit
    fitted_estimator = self._memory_fit(
                       ^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/joblib/memory.py\", line 353, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 68, in _fit_one
    transformer.fit(*args, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/linear_model/_ransac.py\", line 330, in fit
    self._validate_params()
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/base.py\", line 600, in _validate_params
    validate_parameter_constraints(
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/utils/_param_validation.py\", line 97, in validate_parameter_constraints
    raise InvalidParameterError(
sklearn.utils._param_validation.InvalidParameterError: The 'loss' parameter of RANSACRegressor must be a str among {'squared_error', 'absolute_error'} or a callable. Got 'absolute_loss' instead.

--------------------------------------------------------------------------------
13 fits failed with the following error:
Traceback (most recent call last):
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_validation.py\", line 686, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 275, in fit
    fitted_estimator = self._memory_fit(
                       ^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/joblib/memory.py\", line 353, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 68, in _fit_one
    transformer.fit(*args, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/linear_model/_ransac.py\", line 330, in fit
    self._validate_params()
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/base.py\", line 600, in _validate_params
    validate_parameter_constraints(
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/utils/_param_validation.py\", line 97, in validate_parameter_constraints
    raise InvalidParameterError(
sklearn.utils._param_validation.InvalidParameterError: The 'loss' parameter of RANSACRegressor must be a str among {'absolute_error', 'squared_error'} or a callable. Got 'squared_loss' instead.

--------------------------------------------------------------------------------
37 fits failed with the following error:
Traceback (most recent call last):
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_validation.py\", line 686, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 275, in fit
    fitted_estimator = self._memory_fit(
                       ^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/joblib/memory.py\", line 353, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 68, in _fit_one
    transformer.fit(*args, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/linear_model/_ransac.py\", line 330, in fit
    self._validate_params()
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/base.py\", line 600, in _validate_params
    validate_parameter_constraints(
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/utils/_param_validation.py\", line 97, in validate_parameter_constraints
    raise InvalidParameterError(
sklearn.utils._param_validation.InvalidParameterError: The 'loss' parameter of RANSACRegressor must be a str among {'squared_error', 'absolute_error'} or a callable. Got 'squared_loss' instead.
",
	"stack": "---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[13], line 16
      9 exp.setup(
     10     data=data,
     11     target=\"charges\",
     12     transform_target=True,
     13     transformation=True,
     14 )
     15 ransac = exp.create_model(\"ransac\")
---> 16 tuned = exp.tune_model(
     17     estimator=ransac,
     18     choose_better=True,
     19     optimize=sort_metric,
     20 )

File /workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/regression/oop.py:1499, in RegressionExperiment.tune_model(self, estimator, fold, round, n_iter, custom_grid, optimize, custom_scorer, search_library, search_algorithm, early_stopping, early_stopping_max_iters, choose_better, fit_kwargs, groups, return_tuner, verbose, tuner_verbose, return_train_score, **kwargs)
   1307 def tune_model(
   1308     self,
   1309     estimator,
   (...)
   1327     **kwargs,
   1328 ):
   1329     \"\"\"
   1330     This function tunes the hyperparameters of a given estimator. The output of
   1331     this function is a score grid with CV scores by fold of the best selected
   (...)
   1496 
   1497     \"\"\"
-> 1499     return super().tune_model(
   1500         estimator=estimator,
   1501         fold=fold,
   1502         round=round,
   1503         n_iter=n_iter,
   1504         custom_grid=custom_grid,
   1505         optimize=optimize,
   1506         custom_scorer=custom_scorer,
   1507         search_library=search_library,
   1508         search_algorithm=search_algorithm,
   1509         early_stopping=early_stopping,
   1510         early_stopping_max_iters=early_stopping_max_iters,
   1511         choose_better=choose_better,
   1512         fit_kwargs=fit_kwargs,
   1513         groups=groups,
   1514         return_tuner=return_tuner,
   1515         verbose=verbose,
   1516         tuner_verbose=tuner_verbose,
   1517         return_train_score=return_train_score,
   1518         **kwargs,
   1519     )

File /workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pycaret_experiment/supervised_experiment.py:2672, in _SupervisedExperiment.tune_model(self, estimator, fold, round, n_iter, custom_grid, optimize, custom_scorer, search_library, search_algorithm, early_stopping, early_stopping_max_iters, choose_better, fit_kwargs, groups, return_tuner, verbose, tuner_verbose, return_train_score, **kwargs)
   2664     with patch(
   2665         \"sklearn.model_selection._search.sample_without_replacement\",
   2666         pycaret.internal.patches.sklearn._mp_sample_without_replacement,
   2667     ):
   2668         with patch(
   2669             \"sklearn.model_selection._search.ParameterGrid.__getitem__\",
   2670             pycaret.internal.patches.sklearn._mp_ParameterGrid_getitem,
   2671         ):
-> 2672             model_grid.fit(data_X, data_y, groups=groups, **fit_kwargs)
   2673 else:
   2674     model_grid.fit(data_X, data_y, groups=groups, **fit_kwargs)

File /workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_search.py:874, in BaseSearchCV.fit(self, X, y, groups, **fit_params)
    868     results = self._format_results(
    869         all_candidate_params, n_splits, all_out, all_more_results
    870     )
    872     return results
--> 874 self._run_search(evaluate_candidates)
    876 # multimetric is determined here because in the case of a callable
    877 # self.scoring the return type is only known after calling
    878 first_test_score = all_out[0][\"test_scores\"]

File /workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_search.py:1768, in RandomizedSearchCV._run_search(self, evaluate_candidates)
   1766 def _run_search(self, evaluate_candidates):
   1767     \"\"\"Search n_iter candidates from param_distributions\"\"\"
-> 1768     evaluate_candidates(
   1769         ParameterSampler(
   1770             self.param_distributions, self.n_iter, random_state=self.random_state
   1771         )
   1772     )

File /workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_search.py:851, in BaseSearchCV.fit.<locals>.evaluate_candidates(candidate_params, cv, more_results)
    844 elif len(out) != n_candidates * n_splits:
    845     raise ValueError(
    846         \"cv.split and cv.get_n_splits returned \"
    847         \"inconsistent results. Expected {} \"
    848         \"splits, got {}\".format(n_splits, len(out) // n_candidates)
    849     )
--> 851 _warn_or_raise_about_fit_failures(out, self.error_score)
    853 # For callable self.scoring, the return type is only know after
    854 # calling. If the return type is a dictionary, the error scores
    855 # can now be inserted with the correct key. The type checking
    856 # of out will be done in `_insert_error_scores`.
    857 if callable(self.scoring):

File /workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_validation.py:367, in _warn_or_raise_about_fit_failures(results, error_score)
    360 if num_failed_fits == num_fits:
    361     all_fits_failed_message = (
    362         f\"\
All the {num_fits} fits failed.\
\"
    363         \"It is very likely that your model is misconfigured.\
\"
    364         \"You can try to debug the error by setting error_score='raise'.\
\
\"
    365         f\"Below are more details about the failures:\
{fit_errors_summary}\"
    366     )
--> 367     raise ValueError(all_fits_failed_message)
    369 else:
    370     some_fits_failed_message = (
    371         f\"\
{num_failed_fits} fits failed out of a total of {num_fits}.\
\"
    372         \"The score on these train-test partitions for these parameters\"
   (...)
    376         f\"Below are more details about the failures:\
{fit_errors_summary}\"
    377     )

ValueError: 
All the 100 fits failed.
It is very likely that your model is misconfigured.
You can try to debug the error by setting error_score='raise'.

Below are more details about the failures:
--------------------------------------------------------------------------------
26 fits failed with the following error:
Traceback (most recent call last):
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_validation.py\", line 686, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 275, in fit
    fitted_estimator = self._memory_fit(
                       ^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/joblib/memory.py\", line 353, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 68, in _fit_one
    transformer.fit(*args, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/linear_model/_ransac.py\", line 330, in fit
    self._validate_params()
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/base.py\", line 600, in _validate_params
    validate_parameter_constraints(
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/utils/_param_validation.py\", line 97, in validate_parameter_constraints
    raise InvalidParameterError(
sklearn.utils._param_validation.InvalidParameterError: The 'loss' parameter of RANSACRegressor must be a str among {'absolute_error', 'squared_error'} or a callable. Got 'absolute_loss' instead.

--------------------------------------------------------------------------------
24 fits failed with the following error:
Traceback (most recent call last):
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_validation.py\", line 686, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 275, in fit
    fitted_estimator = self._memory_fit(
                       ^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/joblib/memory.py\", line 353, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 68, in _fit_one
    transformer.fit(*args, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/linear_model/_ransac.py\", line 330, in fit
    self._validate_params()
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/base.py\", line 600, in _validate_params
    validate_parameter_constraints(
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/utils/_param_validation.py\", line 97, in validate_parameter_constraints
    raise InvalidParameterError(
sklearn.utils._param_validation.InvalidParameterError: The 'loss' parameter of RANSACRegressor must be a str among {'squared_error', 'absolute_error'} or a callable. Got 'absolute_loss' instead.

--------------------------------------------------------------------------------
13 fits failed with the following error:
Traceback (most recent call last):
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_validation.py\", line 686, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 275, in fit
    fitted_estimator = self._memory_fit(
                       ^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/joblib/memory.py\", line 353, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 68, in _fit_one
    transformer.fit(*args, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/linear_model/_ransac.py\", line 330, in fit
    self._validate_params()
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/base.py\", line 600, in _validate_params
    validate_parameter_constraints(
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/utils/_param_validation.py\", line 97, in validate_parameter_constraints
    raise InvalidParameterError(
sklearn.utils._param_validation.InvalidParameterError: The 'loss' parameter of RANSACRegressor must be a str among {'absolute_error', 'squared_error'} or a callable. Got 'squared_loss' instead.

--------------------------------------------------------------------------------
37 fits failed with the following error:
Traceback (most recent call last):
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/model_selection/_validation.py\", line 686, in _fit_and_score
    estimator.fit(X_train, y_train, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 275, in fit
    fitted_estimator = self._memory_fit(
                       ^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/joblib/memory.py\", line 353, in __call__
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/pycaret/internal/pipeline.py\", line 68, in _fit_one
    transformer.fit(*args, **fit_params)
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/linear_model/_ransac.py\", line 330, in fit
    self._validate_params()
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/base.py\", line 600, in _validate_params
    validate_parameter_constraints(
  File \"/workspaces/bebefam-exploration/.venv/lib/python3.11/site-packages/sklearn/utils/_param_validation.py\", line 97, in validate_parameter_constraints
    raise InvalidParameterError(
sklearn.utils._param_validation.InvalidParameterError: The 'loss' parameter of RANSACRegressor must be a str among {'squared_error', 'absolute_error'} or a callable. Got 'squared_loss' instead.
"
}

Installed Versions

>>> from pycaret import show_versions; show_versions()

System:
python: 3.11.7 (main, Dec 5 2023, 18:55:16) [GCC 12.2.0]
executable: /workspaces/bebefam-exploration/.venv/bin/python
machine: Linux-6.5.0-14-generic-x86_64-with-glibc2.36

PyCaret required dependencies:
pip: 23.3.2
setuptools: 69.0.3
pycaret: 3.2.0
IPython: 8.20.0
ipywidgets: 8.1.1
tqdm: 4.66.1
numpy: 1.25.2
pandas: 1.5.3
jinja2: 3.1.2
scipy: 1.10.1
joblib: 1.3.2
sklearn: 1.2.2
pyod: 1.1.2
imblearn: 0.11.0
category_encoders: 2.6.3
lightgbm: 4.2.0
numba: 0.58.1
requests: 2.31.0
matplotlib: 3.6.0
scikitplot: 0.3.7
yellowbrick: 1.5
plotly: 5.18.0
plotly-resampler: Not installed
kaleido: 0.2.1
schemdraw: 0.15
statsmodels: 0.14.1
sktime: 0.21.1
tbats: 1.1.3
pmdarima: 2.0.4
psutil: 5.9.7
markupsafe: 2.1.3
pickle5: Not installed
cloudpickle: 3.0.0
deprecation: 2.1.0
xxhash: 3.4.1
wurlitzer: 3.0.3

PyCaret optional dependencies:
shap: 0.44.0
interpret: Not installed
umap: Not installed
ydata_profiling: Not installed
explainerdashboard: Not installed
autoviz: Not installed
fairlearn: Not installed
deepchecks: Not installed
xgboost: 2.0.3
catboost: 1.2.2
kmodes: 0.12.2
mlxtend: 0.23.0
statsforecast: 1.5.0
tune_sklearn: Not installed
ray: Not installed
hyperopt: Not installed
optuna: Not installed
skopt: Not installed
mlflow: 1.27.0
gradio: 3.50.2
fastapi: 0.104.1
uvicorn: 0.25.0
m2cgen: 0.10.0
evidently: 0.2.8
fugue: 0.8.7
streamlit: Not installed
prophet: Not installed

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions