Skip to content

"RuntimeError: Attempt to pop from an empty stack" is raised when running models fit in parallel with threads. #2620

@arik-v

Description

@arik-v

Problem: "RuntimeError: Attempt to pop from an empty stack" is raised when running models fit in parallel with threads.
Similar to: #1855
The custom logger stack is not thread safe and raises the RuntimeError.

This does not happen in version 1.2.2, the code finishes successfully.

How to reproduce:

import numpy as np
from joblib import Parallel, delayed
from catboost import CatBoostRegressor

rng = np.random.default_rng(seed = 123)
X = rng.standard_normal(size=(1000, 10))
coef = rng.standard_normal(size=(10, 1))
y = (X @ coef).reshape(-1) + rng.standard_normal(size=1000)

def fit_model(X, y):
    model = CatBoostRegressor(silent=True)
    model.fit(X, y)
    return model

n_models = 10
models = Parallel(n_jobs=n_models, verbose=0,prefer="threads")  (delayed(fit_model)(X, y) for i in range(n_models))

catboost version: 1.2.3
Operating System: Amazon Linux 2023.4.20240319
CPU: Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
CPU(s): 4
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz

GPU: none

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions