[BUG] compare_models doesn't raise an error when errors='raise' and model fails #3738
Description
pycaret version checks
-
I have checked that this issue has not already been reported here.
-
I have confirmed this bug exists on the latest version of pycaret.
-
I have confirmed this bug exists on the master branch of pycaret (pip install -U git+https://github.com/pycaret/pycaret.git@master).
Issue Description
I am running compare_models, including my custom model, and the comparison table has zero for MAE, MAPE, etc., for my custom model. I have some checks inside the custom model to detect problematic data passed into the model, and I raise an error. Even though I have set the errors parameter of compare_models to raise, it doesn't raise the error and produces zero for metric values. For reproducibility of the problem, I am providing dummy data and a regressor, which raises an error when the predict method is called. Pycaret produces zeros for MAE, MAPE, RMSE, etc.
Reproducible Example
from sklearn.base import BaseEstimator, RegressorMixin
import pandas as pd
from pycaret.regression import RegressionExperiment
from sklearn.datasets import make_regression
class DummyRegressor(BaseEstimator, RegressorMixin):
def fit(self, X, y):
return self
def predict(self, X):
raise Exception('DummyEstimator cannot predict')
X, y = make_regression(n_samples=1000, n_features=20, n_informative=10, random_state=42)
data = pd.DataFrame(X)
data['target'] = y
reg_exp = RegressionExperiment()
reg_exp.setup(
data=data,
target='target',
data_split_shuffle = False,
feature_selection = True,
preprocess = True,
session_id = 42,
n_features_to_select = 0.2,
)
best_model = reg_exp.compare_models(
include=[DummyRegressor()],
errors='raise',
)Expected Behavior
should raise an error with message: DummyEstimator cannot predict
Actual Results
Model | MAE | MSE | RMSE | R2 | RMSLE | MAPE | TT (Sec)
DummyRegressor | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1990
Installed Versions
PyCaret required dependencies:
pip: 20.3.4
setuptools: 44.1.1
pycaret: 3.1.0
IPython: 8.5.0
ipywidgets: 7.7.2
tqdm: 4.64.1
numpy: 1.23.3
pandas: 1.5.0
jinja2: 3.1.2
scipy: 1.10.1
joblib: 1.2.0
sklearn: 1.2.2
pyod: 1.1.0
imblearn: 0.11.0
category_encoders: 2.6.2
lightgbm: 3.3.2
numba: 0.57.1
requests: 2.31.0
matplotlib: 3.7.2
scikitplot: 0.3.7
yellowbrick: 1.5
plotly: 5.16.1
plotly-resampler: Not installed
kaleido: 0.2.1
schemdraw: 0.15
statsmodels: 0.14.0
sktime: 0.21.1
tbats: 1.1.3
pmdarima: 2.0.3
psutil: 5.9.2
markupsafe: 2.1.1
pickle5: Not installed
cloudpickle: 2.2.1
deprecation: 2.1.0
xxhash: 3.3.0
wurlitzer: 3.0.3
PyCaret optional dependencies:
shap: Not installed
interpret: Not installed
umap: Not installed
ydata_profiling: Not installed
explainerdashboard: Not installed
autoviz: Not installed
fairlearn: Not installed
deepchecks: Not installed
xgboost: 1.7.6
catboost: 1.2.1
kmodes: 0.12.2
mlxtend: 0.21.0
statsforecast: 1.6.0
tune_sklearn: Not installed
ray: Not installed
hyperopt: Not installed
optuna: 3.2.0
skopt: Not installed
mlflow: Not installed
gradio: Not installed
fastapi: Not installed
uvicorn: Not installed
m2cgen: Not installed
evidently: Not installed
fugue: 0.8.6
streamlit: Not installed
prophet: 1.1.4