2025-01-20 15:05:45,008	ERROR serialization.py:462 -- Failed to unpickle serialized exception
Traceback (most recent call last):
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 51](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/exceptions.py#line=50), in from_ray_exception
    return pickle.loads(ray_exception.serialized_exception)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_catboost'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\_private\serialization.py", line 460](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/_private/serialization.py#line=459), in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\_private\serialization.py", line 342](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/_private/serialization.py#line=341), in _deserialize_object
    return RayError.from_bytes(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 45](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/exceptions.py#line=44), in from_bytes
    return RayError.from_ray_exception(ray_exception)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 54](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/exceptions.py#line=53), in from_ray_exception
    raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception
	Warning: Exception caused CatBoost_BAG_L1 to fail during training... Skipping this model.
		System error: Failed to unpickle serialized exception
traceback: Traceback (most recent call last):
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 51](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/exceptions.py#line=50), in from_ray_exception
    return pickle.loads(ray_exception.serialized_exception)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_catboost'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\_private\serialization.py", line 460](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/_private/serialization.py#line=459), in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\_private\serialization.py", line 342](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/_private/serialization.py#line=341), in _deserialize_object
    return RayError.from_bytes(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 45](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/exceptions.py#line=44), in from_bytes
    return RayError.from_ray_exception(ray_exception)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 54](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/exceptions.py#line=53), in from_ray_exception
    raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception

Detailed Traceback:
Traceback (most recent call last):
  File "[C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\trainer\abstract_trainer.py", line 2106](file:///C:/Users/celes/anaconda3/Lib/site-packages/autogluon/core/trainer/abstract_trainer.py#line=2105), in _train_and_save
    model = self._train_single(**model_fit_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\trainer\abstract_trainer.py", line 1993](file:///C:/Users/celes/anaconda3/Lib/site-packages/autogluon/core/trainer/abstract_trainer.py#line=1992), in _train_single
    model = model.fit(X=X, y=y, X_val=X_val, y_val=y_val, X_test=X_test, y_test=y_test, total_resources=total_resources, **model_fit_kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\abstract\abstract_model.py", line 925](file:///C:/Users/celes/anaconda3/Lib/site-packages/autogluon/core/models/abstract/abstract_model.py#line=924), in fit
    out = self._fit(**kwargs)
          ^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\stacker_ensemble_model.py", line 270](file:///C:/Users/celes/anaconda3/Lib/site-packages/autogluon/core/models/ensemble/stacker_ensemble_model.py#line=269), in _fit
    return super()._fit(X=X, y=y, time_limit=time_limit, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\bagged_ensemble_model.py", line 298](file:///C:/Users/celes/anaconda3/Lib/site-packages/autogluon/core/models/ensemble/bagged_ensemble_model.py#line=297), in _fit
    self._fit_folds(
  File "[C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\bagged_ensemble_model.py", line 724](file:///C:/Users/celes/anaconda3/Lib/site-packages/autogluon/core/models/ensemble/bagged_ensemble_model.py#line=723), in _fit_folds
    fold_fitting_strategy.after_all_folds_scheduled()
  File "[C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 690](file:///C:/Users/celes/anaconda3/Lib/site-packages/autogluon/core/models/ensemble/fold_fitting_strategy.py#line=689), in after_all_folds_scheduled
    self._run_parallel(X, y, X_pseudo, y_pseudo, model_base_ref, time_limit_fold, head_node_id)
  File "[C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 631](file:///C:/Users/celes/anaconda3/Lib/site-packages/autogluon/core/models/ensemble/fold_fitting_strategy.py#line=630), in _run_parallel
    self._process_fold_results(finished, unfinished, fold_ctx)
  File "[C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 587](file:///C:/Users/celes/anaconda3/Lib/site-packages/autogluon/core/models/ensemble/fold_fitting_strategy.py#line=586), in _process_fold_results
    raise processed_exception
  File "[C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 550](file:///C:/Users/celes/anaconda3/Lib/site-packages/autogluon/core/models/ensemble/fold_fitting_strategy.py#line=549), in _process_fold_results
    fold_model, pred_proba, time_start_fit, time_end_fit, predict_time, predict_1_time, predict_n_size, fit_num_cpus, fit_num_gpus = self.ray.get(finished)
                                                                                                                                     ^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\_private\auto_init_hook.py", line 21](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/_private/auto_init_hook.py#line=20), in auto_init_wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\_private\client_mode_hook.py", line 103](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/_private/client_mode_hook.py#line=102), in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\_private\worker.py", line 2772](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/_private/worker.py#line=2771), in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\_private\worker.py", line 921](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/_private/worker.py#line=920), in get_objects
    raise value
ray.exceptions.RaySystemError: System error: Failed to unpickle serialized exception
traceback: Traceback (most recent call last):
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 51](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/exceptions.py#line=50), in from_ray_exception
    return pickle.loads(ray_exception.serialized_exception)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_catboost'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\_private\serialization.py", line 460](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/_private/serialization.py#line=459), in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\_private\serialization.py", line 342](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/_private/serialization.py#line=341), in _deserialize_object
    return RayError.from_bytes(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 45](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/exceptions.py#line=44), in from_bytes
    return RayError.from_ray_exception(ray_exception)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "[C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 54](file:///C:/Users/celes/anaconda3/Lib/site-packages/ray/exceptions.py#line=53), in from_ray_exception
    raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception