[BUG] Dask RandomForestClassifier get_combined_model() and .internal_model return None #3331
Description
Describe the bug
I've been trying out cuml.dask.ensemble.RandomForestClassifier for multi-node, multi-GPU training. Training works without issue and the results look reasonable, but I'm unable to save the trained model using the public API of cuml.dask.ensemble.RandomForestClassifier.
I see the same behavior on single-node, single-GPU training with dask_cuda.LocalCUDACluster.
Since these model objects cannot be pickled directly, and since the steps at https://docs.rapids.ai/api/cuml/stable/pickling_cuml_models.html#Distributed-Model-Pickling aren't working for me, I'm not sure how to save my trained model.
Thanks for your time and consideration!
Steps/Code to reproduce bug
from dask.distributed import Client
from dask_cuda import LocalCUDACluster
cluster = LocalCUDACluster()
client = Client(cluster)
import dask_cudf
df = dask_cudf.read_csv(
's3://nyc-tlc/trip data/yellow_tripdata_2019-01.csv',
parse_dates=['tpep_pickup_datetime', 'tpep_dropoff_datetime'],
storage_options={'anon': True},
assume_missing=True,
)
target_col = 'high_tip'
df = df[df.fare_amount > 0] # avoid divide-by-zero
df['tip_fraction'] = df.tip_amount / df.fare_amount
df[target_col] = (df['tip_fraction'] > 0.2)
df['pickup_weekday'] = df.tpep_pickup_datetime.dt.weekday
df['pickup_hour'] = df.tpep_pickup_datetime.dt.hour
df['pickup_week_hour'] = (df.pickup_weekday * 24) + df.pickup_hour
df['pickup_minute'] = df.tpep_pickup_datetime.dt.minute
features = [c for c in df.columns if c.startswith('pickup_')]
df = df[features + [target_col]].astype('float32').fillna(-1)
df[target_col] = df[target_col].astype('int32')
from cuml.dask.ensemble import RandomForestClassifier
rfc = RandomForestClassifier(
n_estimators=10,
max_depth=5
)
rfc.fit(df[features], df[target_col])
# returns None
rfc.get_combined_model()
# returns None
rfc.internal_modelExpected behavior
Based on https://docs.rapids.ai/api/cuml/stable/pickling_cuml_models.html#Distributed-Model-Pickling, I expected that after .fit() completes, I could call .get_combined_model() to produce a single-GPU model object which could be pickled.
However, after calling .fit(), I see that the model object's .internal_model is None, as is the result of .get_combined_model(). When I run the example at https://docs.rapids.ai/api/cuml/stable/pickling_cuml_models.html#Distributed-Model-Pickling (which uses K-means), it does produce the expected model object. So maybe this issue is specific to RandomForestClassifier.
Environment details (please complete the following information):
- Environment location:
- docker, with image
saturncloud/saturn-gpu:2020.11.30 - the only thing I installed at runtime is
gpustat, withpip install gpustat
- docker, with image
- Linux Distro/Architecture:
- Ubuntu 18.04.5 LTS
- GPU Model/Driver: [V100 and driver 396.44]
- Tesla V4
- driver: 450.51.06
- CUDA:
- 11.0
- Method of cuDF & cuML install: conda
result of running 'conda list' (click me)
# packages in environment at /srv/conda/envs/saturn:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
_tflow_select 2.1.0 gpu defaults
abseil-cpp 20200225.2 he1b5a44_2 conda-forge
absl-py 0.11.0 py37h06a4308_0 defaults
aiobotocore 1.1.2 py_0 defaults
aiohttp 3.7.2 py37h27cfd23_1 defaults
aioitertools 0.7.0 py_0 defaults
appdirs 1.4.4 py_0 defaults
argon2-cffi 20.1.0 py37h7b6447c_1 defaults
arrow-cpp 0.17.1 py37h1234567_11_cuda conda-forge
arrow-cpp-proc 2.0.0 cuda conda-forge
asn1crypto 1.4.0 py_0 defaults
astor 0.8.1 py37_0 defaults
async-timeout 3.0.1 py37_0 defaults
async_generator 1.10 py37h28b3542_0 defaults
atk 2.36.0 0 defaults
atk-1.0 2.36.0 haf93ef1_1 conda-forge
attrs 20.3.0 pyhd3eb1b0_0 defaults
aws-sdk-cpp 1.7.164 hc831370_1 conda-forge
azure-common 1.1.24 py_0 conda-forge
azure-core 1.9.0 pyhd3deb0d_0 conda-forge
azure-nspkg 3.0.2 py_0 conda-forge
azure-storage-blob 12.6.0 pyhd3deb0d_0 conda-forge
backcall 0.2.0 py_0 defaults
black 20.8b1 pypi_0 pypi
blas 1.0 mkl defaults
bleach 3.2.1 py_0 defaults
blessings 1.7 pypi_0 pypi
blinker 1.4 py37_0 defaults
bokeh 2.2.3 py37_0 defaults
boost 1.72.0 py37h48f8a5e_1 conda-forge
boost-cpp 1.72.0 h8e57a91_0 conda-forge
boto 2.49.0 py37_0 defaults
boto3 1.14.43 py_0 defaults
botocore 1.17.44 py_0 defaults
brotli 1.0.9 he6710b0_2 defaults
brotli-python 1.0.9 py37heb0550a_2 defaults
brotlipy 0.7.0 py37h27cfd23_1003 defaults
bzip2 1.0.8 h7b6447c_0 defaults
c-ares 1.17.1 h27cfd23_0 defaults
ca-certificates 2020.10.14 0 defaults
cachetools 4.1.1 py_0 defaults
cairo 1.16.0 hcf35c78_1003 conda-forge
certifi 2020.11.8 py37h06a4308_0 defaults
cffi 1.14.0 py37h2e261b9_0 defaults
cfitsio 3.470 hf0d0db6_5 defaults
chardet 3.0.4 py37h06a4308_1003 defaults
click 7.1.2 py_0 defaults
click-plugins 1.1.1 py_0 defaults
cligj 0.7.1 py37h06a4308_0 defaults
cloudpickle 1.6.0 py_0 defaults
colorcet 2.0.2 py_0 defaults
croniter 0.3.35 py_0 defaults
cryptography 2.9.2 py37h1ba5d50_0 defaults
cudatoolkit 10.1.243 h6bb024c_0 nvidia
cudf 0.15.0 cuda_10.1_py37_g71cb8c0e0_0 rapidsai
cudf_kafka 0.15.0 py37_g71cb8c0e0_0 rapidsai
cudnn 7.6.5 cuda10.1_0 defaults
cugraph 0.15.0 py37_gb34091ac_0 rapidsai
cuml 0.15.0 cuda10.1_py37_ga3002e587_0 rapidsai
cupti 10.1.168 0 defaults
cupy 7.8.0 py37h0632833_1 conda-forge
curl 7.69.1 hbc83047_0 defaults
cusignal 0.15.0 py37_gdac6dff_0 rapidsai
cuspatial 0.15.0 py37_gc5b7527_0 rapidsai
custreamz 0.15.0 py37_g71cb8c0e0_0 rapidsai
cuxfilter 0.15.0 py37_gf17ebcb_0 rapidsai
cvxpy 1.1.7 py37h3340039_0 conda-forge
cvxpy-base 1.1.7 py37h9fdb41a_0 conda-forge
cycler 0.10.0 py37_0 defaults
cyrus-sasl 2.1.27 h063b49f_1 conda-forge
cytoolz 0.11.0 py37h7b6447c_0 defaults
dash 1.17.0 pyhd3eb1b0_0 defaults
dash-bootstrap-components 0.10.7 pyh9f0ad1d_0 conda-forge
dash-core-components 1.3.1 py_0 defaults
dash-daq 0.5.0 pyh9f0ad1d_1 conda-forge
dash-html-components 1.0.1 py_0 defaults
dash-renderer 1.1.2 py_0 defaults
dash-table 4.4.1 py_0 defaults
dask 2.30.0 py_0 defaults
dask-core 2.30.0 py_0 defaults
dask-cuda 0.15.0 py37_0 rapidsai
dask-cudf 0.15.0 py37_g71cb8c0e0_0 rapidsai
dask-glm 0.2.0 py37_0 defaults
dask-ml 1.7.0 py_0 defaults
dask-saturn 0.1.3 py_0 saturncloud
dask-xgboost 0.2.0.dev28 cuda10.1py37_0 rapidsai
datashader 0.11.1 py_0 defaults
datashape 0.5.4 py37_1 defaults
dbus 1.13.18 hb2f20db_0 defaults
decorator 4.4.2 py_0 defaults
defusedxml 0.6.0 py_0 defaults
distributed 2.30.0 py37_0 defaults
dlpack 0.3 he6710b0_1 defaults
docker-py 4.4.0 py37h06a4308_4 defaults
docker-pycreds 0.4.0 py_0 defaults
docutils 0.15.2 py37_0 defaults
double-conversion 3.1.5 he6710b0_1 defaults
ecos 2.0.7.post1 py37heb32a55_0 defaults
entrypoints 0.3 py37_0 defaults
expat 2.2.10 he6710b0_2 defaults
faiss-proc 1.0.0 cuda conda-forge
fastavro 1.2.0 py37h27cfd23_0 defaults
fastparquet 0.4.1 py37heb32a55_0 defaults
fastrlock 0.5 py37he6710b0_0 defaults
fiona 1.8.13.post1 py37hc820daa_0 defaults
flake8 3.8.4 pypi_0 pypi
flask 1.1.2 py_0 defaults
flask-compress 1.8.0 pyhd3eb1b0_0 defaults
font-ttf-dejavu-sans-mono 2.37 h6964260_0 defaults
font-ttf-inconsolata 2.001 hcb22688_0 defaults
font-ttf-source-code-pro 2.030 h7457263_0 defaults
font-ttf-ubuntu 0.83 h8b1ccd4_0 defaults
fontconfig 2.13.1 h86ecdb6_1001 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
freetype 2.10.4 h5ab3b9f_0 defaults
freexl 1.0.5 h14c3975_0 defaults
fribidi 1.0.10 h7b6447c_0 defaults
fsspec 0.8.3 py_0 defaults
future 0.18.2 py37_1 defaults
gast 0.2.2 py37_0 defaults
gdal 3.0.4 py37h4b180d9_10 conda-forge
gdk-pixbuf 2.38.2 h3f25603_4 conda-forge
gensim 3.8.0 py37h962f231_0 defaults
geopandas 0.8.1 py_0 defaults
geos 3.8.1 he6710b0_0 defaults
geotiff 1.6.0 h05acad5_0 conda-forge
gettext 0.19.8.1 hd7bead4_3 defaults
gflags 2.2.2 he6710b0_0 defaults
giflib 5.2.1 h7b6447c_0 defaults
glib 2.63.1 h5a9c865_0 defaults
glog 0.4.0 he6710b0_0 defaults
gmp 6.1.2 h6c8ec71_1 defaults
gobject-introspection 1.56.1 py37hbc4ca2d_2 defaults
google-api-core 1.22.2 py37_0 defaults
google-auth 1.23.0 pyhd3eb1b0_0 defaults
google-auth-oauthlib 0.4.2 pyhd3eb1b0_2 defaults
google-cloud-core 1.4.3 pyhd3eb1b0_1 defaults
google-cloud-storage 1.33.0 pyhd3eb1b0_0 defaults
google-crc32c 1.0.0 py37h7b6447c_0 defaults
google-pasta 0.2.0 py_0 defaults
google-resumable-media 1.1.0 py_1 defaults
googleapis-common-protos 1.52.0 py37_0 defaults
gpustat 0.6.0 pypi_0 pypi
graphite2 1.3.14 h23475e2_0 defaults
graphviz 2.42.3 h6939c30_2 conda-forge
grpc-cpp 1.30.2 heedbac9_0 conda-forge
grpcio 1.31.0 py37hf8bcb03_0 defaults
gst-plugins-base 1.14.5 h0935bb2_2 conda-forge
gstreamer 1.14.5 h36ae1b5_2 conda-forge
gtk2 2.24.32 h586f36d_1 conda-forge
gts 0.7.6 h08bb679_0 conda-forge
h5py 2.10.0 py37hd6299e0_1 defaults
harfbuzz 2.4.0 h9f30f68_3 conda-forge
hdf4 4.2.13 h3ca952b_2 defaults
hdf5 1.10.6 nompi_h3c11f04_101 conda-forge
heapdict 1.0.1 py_0 defaults
icu 64.2 he1b5a44_1 conda-forge
idna 2.9 py_1 defaults
importlib-metadata 2.0.0 py_1 defaults
importlib_metadata 2.0.0 1 defaults
importlib_resources 3.3.0 py37h06a4308_0 defaults
intel-openmp 2020.2 254 defaults
ipykernel 5.3.4 py37h5ca1d4c_0 defaults
ipython 7.19.0 py37hb070fc8_0 defaults
ipython_genutils 0.2.0 pyhd3eb1b0_1 defaults
ipywidgets 7.5.1 py_1 defaults
isodate 0.6.0 py_1 defaults
itsdangerous 1.1.0 py37_0 defaults
jedi 0.17.2 py37_0 defaults
jinja2 2.11.2 py_0 defaults
jmespath 0.10.0 py_0 defaults
joblib 0.17.0 py_0 defaults
jpeg 9d h36c2ea0_0 conda-forge
json-c 0.13.1 h1bed415_0 defaults
jsonschema 3.2.0 py_2 defaults
jupyter-server-proxy 1.5.0 py_0 conda-forge
jupyter_client 6.1.7 py_0 defaults
jupyter_core 4.7.0 py37h06a4308_0 defaults
jupyter_server 1.0.7 py37h89c1867_0 conda-forge
jupyterlab_pygments 0.1.2 py_0 defaults
kealib 1.4.13 h33137a7_1 conda-forge
keras-applications 1.0.8 py_1 defaults
keras-preprocessing 1.1.0 py_1 defaults
kiwisolver 1.3.0 py37h2531618_0 defaults
krb5 1.17.1 h173b8e3_0 defaults
lcms2 2.11 h396b838_0 defaults
ld_impl_linux-64 2.33.1 h53a641e_7 defaults
libblas 3.8.0 21_mkl conda-forge
libcrc32c 1.1.1 he6710b0_2 defaults
libcudf 0.15.0 cuda10.1_g71cb8c0e0_0 rapidsai
libcudf_kafka 0.15.0 g71cb8c0e0_0 rapidsai
libcugraph 0.15.0 cuda10.1_gb34091ac_0 rapidsai
libcuml 0.15.0 cuda10.1_ga3002e587_0 rapidsai
libcumlprims 0.15.0 cuda10.1_gdbd0d39_0 nvidia
libcurl 7.69.1 h20c2e04_0 defaults
libcuspatial 0.15.0 cuda10.1_gc5b7527_0 rapidsai
libdap4 3.20.6 h1d1bd15_0 conda-forge
libedit 3.1.20191231 h14c3975_1 defaults
libevent 2.1.10 hcdb4288_3 conda-forge
libfaiss 1.6.3 he68dc02_3_cuda conda-forge
libffi 3.2.1 hf484d3e_1007 defaults
libgcc-ng 9.3.0 h5dbcf3e_17 conda-forge
libgcrypt 1.8.7 h27cfd23_0 defaults
libgdal 3.0.4 he6a97d6_10 conda-forge
libgfortran-ng 7.3.0 hdf63c60_0 defaults
libgomp 9.3.0 h5dbcf3e_17 conda-forge
libgpg-error 1.39 he6710b0_0 defaults
libgsasl 1.8.0 2 conda-forge
libhwloc 2.1.0 h3c4fd83_0 conda-forge
libiconv 1.15 h63c8f33_5 defaults
libkml 1.3.0 hd79254b_1012 conda-forge
liblapack 3.8.0 21_mkl conda-forge
libnetcdf 4.7.4 nompi_h84807e1_104 conda-forge
libntlm 1.5 h7b6447c_0 defaults
libpng 1.6.37 hbc83047_0 defaults
libpq 12.2 h20c2e04_0 defaults
libprotobuf 3.12.4 hd408876_0 defaults
librdkafka 1.4.0 h40bdf00_0 conda-forge
librmm 0.15.0 cuda10.1_g8005ca5_0 rapidsai
libsodium 1.0.18 h7b6447c_0 defaults
libspatialindex 1.9.3 he6710b0_0 defaults
libspatialite 4.3.0a h2482549_1038 conda-forge
libssh2 1.9.0 h1ba5d50_1 defaults
libstdcxx-ng 9.1.0 hdf63c60_0 defaults
libthrift 0.13.0 hbe8ec66_6 conda-forge
libtiff 4.1.0 h2733197_1 defaults
libtool 2.4.6 h7b6447c_1005 defaults
libuuid 2.32.1 h14c3975_1000 conda-forge
libwebp 1.1.0 h76fa15c_4 conda-forge
libwebp-base 1.1.0 h7b6447c_3 defaults
libxcb 1.14 h7b6447c_0 defaults
libxgboost 1.2.0dev.rapidsai0.15 cuda10.1_611 rapidsai
libxml2 2.9.10 hee79883_0 conda-forge
llvmlite 0.35.0rc3 py37hf484d3e_0 numba
locket 0.2.0 py37_1 defaults
lz4-c 1.9.2 heb0550a_3 defaults
markdown 3.3.3 py37h06a4308_0 defaults
markupsafe 1.1.1 py37h14c3975_1 defaults
marshmallow 3.9.1 pyhd3eb1b0_0 defaults
marshmallow-oneofschema 2.0.1 py_0 conda-forge
matplotlib 3.3.2 0 defaults
matplotlib-base 3.3.2 py37h817c723_0 defaults
mccabe 0.6.1 pypi_0 pypi
mistune 0.8.4 py37h14c3975_1001 defaults
mkl 2020.2 256 defaults
mkl-service 2.3.0 py37he904b0f_0 defaults
mkl_fft 1.2.0 py37h23d657b_0 defaults
mkl_random 1.1.1 py37h0573a6f_0 defaults
msgpack-python 1.0.0 py37hfd86e86_1 defaults
msrest 0.6.19 pyh9f0ad1d_0 conda-forge
multidict 4.7.6 py37h7b6447c_1 defaults
multipledispatch 0.6.0 py37_0 defaults
munch 2.5.0 py_0 defaults
mypy_extensions 0.4.3 py37_0 defaults
natsort 7.1.0 pyhd3eb1b0_0 defaults
nbclient 0.5.1 py_0 defaults
nbconvert 6.0.7 py37_0 defaults
nbformat 5.0.8 py_0 defaults
nccl 2.7.8.1 h51cf6c1_1 conda-forge
ncurses 6.2 he6710b0_1 defaults
nest-asyncio 1.4.3 pyhd3eb1b0_0 defaults
ninja 1.10.2 py37hff7bd54_0 defaults
nltk 3.5 py_0 defaults
nodejs 10.13.0 he6710b0_0 defaults
notebook 6.1.4 py37_0 defaults
numba 0.52.0rc3 np1.11py3.7h04863e7_gac5bf3e39_0 numba
numpy 1.19.2 py37h54aff64_0 defaults
numpy-base 1.19.2 py37hfa32c7d_0 defaults
nvidia-ml-py3 7.352.0 pypi_0 pypi
oauthlib 3.1.0 py_0 defaults
olefile 0.46 py37_0 defaults
openjpeg 2.3.1 h981e76c_3 conda-forge
openssl 1.1.1h h7b6447c_0 defaults
opt_einsum 3.1.0 py_0 defaults
oscrypto 1.2.0 py_0 conda-forge
osqp 0.6.1 py37h0da4684_2 conda-forge
packaging 20.4 py_0 defaults
pandas 1.0.5 py37h0573a6f_0 defaults
pandoc 2.11 hb0f4dca_0 defaults
pandocfilters 1.4.3 py37h06a4308_1 defaults
panel 0.10.2 pyhd3eb1b0_0 defaults
pango 1.42.4 h7062337_4 conda-forge
param 1.10.0 pyhd3eb1b0_0 defaults
parquet-cpp 1.5.1 2 conda-forge
parso 0.7.0 py_0 defaults
partd 1.1.0 py_0 defaults
pathspec 0.8.1 pypi_0 pypi
pcre 8.44 he6710b0_0 defaults
pendulum 2.1.2 pyhd3eb1b0_1 defaults
pexpect 4.8.0 pyhd3eb1b0_3 defaults
pickle5 0.0.11 py37h8f50634_0 conda-forge
pickleshare 0.7.5 py37_1001 defaults
pillow 8.0.1 py37he98fc37_0 defaults
pip 20.2.4 py37h06a4308_0 defaults
pixman 0.38.0 h7b6447c_0 defaults
plotly 4.13.0 pyhd3eb1b0_0 defaults
poppler 0.87.0 h4190859_1 conda-forge
poppler-data 0.4.10 h06a4308_0 defaults
postgresql 12.2 h20c2e04_0 defaults
prefect 0.13.18 pyhd8ed1ab_0 conda-forge
proj 7.0.0 h59a7b90_1 defaults
prometheus_client 0.9.0 pyhd3eb1b0_0 defaults
prompt-toolkit 3.0.8 py_0 defaults
protobuf 3.12.4 py37he6710b0_0 defaults
psutil 5.7.2 py37h7b6447c_0 defaults
ptyprocess 0.6.0 pyhd3eb1b0_2 defaults
py-xgboost 1.2.0dev.rapidsai0.15 cuda10.1py37_611 rapidsai
pyarrow 0.17.1 py37h1234567_11_cuda conda-forge
pyasn1 0.4.8 py_0 defaults
pyasn1-modules 0.2.8 py_0 defaults
pycodestyle 2.6.0 pypi_0 pypi
pycparser 2.20 py_2 defaults
pycryptodomex 3.9.9 py37h27cfd23_1 defaults
pyct 0.4.8 py37_0 defaults
pydeck 0.5.0 pyh9f0ad1d_0 conda-forge
pyee 7.0.4 pyh9f0ad1d_0 conda-forge
pyflakes 2.2.0 pypi_0 pypi
pygments 2.7.2 pyhd3eb1b0_0 defaults
pyjwt 1.7.1 py37_0 defaults
pynvml 8.0.4 py_1 conda-forge
pyopenssl 20.0.0 pyhd3eb1b0_1 defaults
pyparsing 2.4.7 py_0 defaults
pyppeteer 0.2.2 py_1 conda-forge
pyproj 2.6.1.post1 py37h34dd122_0 conda-forge
pyqt 5.9.2 py37h05f1152_2 defaults
pyrsistent 0.17.3 py37h7b6447c_0 defaults
pysocks 1.7.1 py37_1 defaults
python 3.7.7 hcf32534_0_cpython defaults
python-box 5.2.0 pyhd8ed1ab_0 conda-forge
python-confluent-kafka 1.3.0 py37h8f50634_1 conda-forge
python-dateutil 2.8.1 py_0 defaults
python-slugify 4.0.1 py_0 defaults
python_abi 3.7 1_cp37m conda-forge
pytorch 1.4.0 py3.7_cuda10.1.243_cudnn7.6.3_0 pytorch
pytz 2020.4 pyhd3eb1b0_0 defaults
pytzdata 2020.1 py_0 defaults
pyviz_comms 0.7.6 py_0 defaults
pyyaml 5.3.1 py37h7b6447c_1 defaults
pyzmq 20.0.0 py37h2531618_1 defaults
qt 5.9.7 h0c104cb_3 conda-forge
rapids 0.15.1 cuda10.1_py37_gc1db54b_5 rapidsai
rapids-xgboost 0.15.1 cuda10.1_py37_gc1db54b_5 rapidsai
re2 2020.07.06 he1b5a44_1 conda-forge
readline 8.0 h7b6447c_0 defaults
regex 2020.11.13 py37h27cfd23_0 defaults
requests 2.23.0 py37_0 defaults
requests-oauthlib 1.3.0 py_0 defaults
retrying 1.3.3 py37_2 defaults
rmm 0.15.0 cuda_10.1_py37_g8005ca5_0 rapidsai
rsa 4.6 py_0 defaults
rtree 0.9.4 py37_1 defaults
ruamel.yaml 0.16.12 py37h7b6447c_1 defaults
ruamel.yaml.clib 0.2.2 py37h7b6447c_0 defaults
s3fs 0.5.1 py_0 defaults
s3transfer 0.3.3 py37_1 defaults
scikit-learn 0.23.2 py37h0573a6f_0 defaults
scipy 1.5.2 py37h0b6359f_0 defaults
scs 2.1.2 py37h26cea63_2 conda-forge
send2trash 1.5.0 py37_0 defaults
setuptools 50.3.1 py37h06a4308_1 defaults
shapely 1.7.1 py37hedb1597_1 conda-forge
simpervisor 0.3 py_1 conda-forge
sip 4.19.8 py37hf484d3e_0 defaults
six 1.15.0 py37h06a4308_0 defaults
smart_open 3.0.0 py_0 defaults
snappy 1.1.8 he6710b0_0 defaults
snowflake-connector-python 2.3.2 py37h336dea5_0 conda-forge
snowflake-sqlalchemy 1.2.4 pyh9f0ad1d_0 conda-forge
sortedcontainers 2.2.2 py_0 defaults
spdlog 1.8.1 h7739ffd_0 defaults
sqlalchemy 1.3.20 py37h27cfd23_0 defaults
sqlite 3.33.0 h62c20be_0 defaults
streamz 0.6.0 py_0 defaults
tabulate 0.8.7 py37_0 defaults
tbb 2020.3 hfd86e86_0 defaults
tblib 1.7.0 py_0 defaults
tensorboard 2.1.0 py3_0 defaults
tensorflow 2.1.0 gpu_py37h7a4bb67_0 defaults
tensorflow-base 2.1.0 gpu_py37h6c5654b_0 defaults
tensorflow-estimator 2.1.0 pyhd54b08b_0 defaults
termcolor 1.1.0 py37_1 defaults
terminado 0.9.1 py37_0 defaults
testpath 0.4.4 py_0 defaults
text-unidecode 1.3 py_0 defaults
threadpoolctl 2.1.0 pyh5ca1d4c_0 defaults
thrift 0.11.0 py37hf484d3e_0 defaults
thrift-compiler 0.13.0 hbe8ec66_6 conda-forge
thrift-cpp 0.13.0 6 conda-forge
tiledb 1.7.7 h8efa9f0_3 conda-forge
tk 8.6.10 hbc83047_0 defaults
toml 0.10.1 py_0 defaults
toolz 0.11.1 py_0 defaults
torchvision 0.5.0 py37_cu101 pytorch
tornado 6.0.4 py37h7b6447c_1 defaults
tqdm 4.51.0 pyhd3eb1b0_0 defaults
traitlets 5.0.5 py_0 defaults
treelite 0.92 py37h023e13c_2 conda-forge
treelite-runtime 0.92 pypi_0 pypi
typed-ast 1.4.1 pypi_0 pypi
typing-extensions 3.7.4.3 0 defaults
typing_extensions 3.7.4.3 py_0 defaults
ucx 1.8.1+g6b29558 cuda10.1_0 rapidsai
ucx-py 0.15.0+g6b29558 py37_0 rapidsai
unidecode 1.1.1 py_0 defaults
urllib3 1.25.11 py_0 defaults
voila 0.2.4 py_0 conda-forge
wcwidth 0.2.5 py_0 defaults
webencodings 0.5.1 py37_1 defaults
websocket-client 0.57.0 py37_2 defaults
websockets 8.1 py37h8f50634_2 conda-forge
werkzeug 1.0.1 py_0 defaults
wheel 0.35.1 pyhd3eb1b0_0 defaults
widgetsnbextension 3.5.1 py37_0 defaults
wordcloud 1.8.1 py37h4abf009_1 conda-forge
wrapt 1.12.1 py37h7b6447c_1 defaults
xarray 0.16.1 py_0 defaults
xerces-c 3.2.2 h8412b87_1004 conda-forge
xgboost 1.2.0dev.rapidsai0.15 cuda10.1py37_611 rapidsai
xorg-kbproto 1.0.7 h14c3975_1002 conda-forge
xorg-libice 1.0.10 h516909a_0 conda-forge
xorg-libsm 1.2.3 h84519dc_1000 conda-forge
xorg-libx11 1.6.12 h516909a_0 conda-forge
xorg-libxau 1.0.9 h14c3975_0 conda-forge
xorg-libxext 1.3.4 h516909a_0 conda-forge
xorg-libxpm 3.5.13 h516909a_0 conda-forge
xorg-libxrender 0.9.10 h516909a_1002 conda-forge
xorg-libxt 1.1.5 h516909a_1003 conda-forge
xorg-renderproto 0.11.1 h14c3975_1002 conda-forge
xorg-xextproto 7.3.0 h14c3975_1002 conda-forge
xorg-xproto 7.0.31 h14c3975_1007 conda-forge
xz 5.2.5 h7b6447c_0 defaults
yaml 0.2.5 h7b6447c_0 defaults
yarl 1.5.1 py37h7b6447c_0 defaults
zeromq 4.3.3 he6710b0_3 defaults
zict 2.0.0 py_0 defaults
zipp 3.4.0 pyhd3eb1b0_0 defaults
zlib 1.2.11 h7b6447c_3 defaults
zstd 1.4.5 h9ceee32_0 defaults
result of running 'conda info' (click me)
active environment : /opt/conda/envs/saturn
active env location : /opt/conda/envs/saturn
user config file : /home/jovyan/.condarc
populated config files : /srv/conda/.condarc
conda version : 4.8.2
conda-build version : not installed
python version : 3.7.7.final.0
virtual packages : __cuda=11.0
__glibc=2.27
base environment : /srv/conda (writable)
channel URLs : https://conda.saturncloud.io/pkgs/linux-64
https://conda.saturncloud.io/pkgs/noarch
https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /srv/conda/pkgs
/home/jovyan/.conda/pkgs
envs directories : /srv/conda/envs
/home/jovyan/.conda/envs
platform : linux-64
user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.7 Linux/4.14.203-156.332.amzn2.x86_64 ubuntu/18.04.5 glibc/2.27
UID:GID : 1000:100
netrc file : None
offline mode : False
Additional context
I originally observed this behavior using a dask-kubernetes cluster on EKS. The code above using LocalCUDACluster was the smallest reproducible example I could produce.
I observed this behavior using the managed Jupyter instances available from Saturn Cloud. I have no reason to believe that something Saturn-specific is causing this issue, but if reviewers are unable to reproduce it I'll try on a personal machine.