Spacy fails with "module 'cupy' has no attribute 'cublas'" #920
Description
How to reproduce the behaviour
- Create a new venv
- Install Spacy with cupy, just follow Spacy doc for conda
- Try to train a model with GPU
Your Environment
- Operating System: Ubuntu 22.04
- Python Version Used: 3.11
- Thinc Version Used: 8.2.2
- Environment Information:
$ micromamba list | grep thinc
thinc 8.2.2 py311h92ebd52_0 conda-forge
$ micromamba list | grep cupy
cupy 13.0.0 py311h878bca4_1 conda-forge
cupy-core 13.0.0 py311heecd119_1 conda-forge
$ micromamba list | grep cuda
cuda-version 11.8 h70ddcb2_2 conda-forge
cudatoolkit 11.8.0 h4ba93d1_12 conda-forge
$ spacy info
============================== Info about spaCy ==============================
spaCy version 3.7.2
Location /home/vitaly/micromamba/envs/pc/lib/python3.11/site-packages/spacy
Platform Linux-6.5.0-14-generic-x86_64-with-glibc2.35
Python version 3.11.7
Pipelines
Error
Spacy fails with the error stack as below:
File "/home/vitaly/micromamba/envs/pc/lib/python3.11/site-packages/thinc/backends/cupy_ops.py", line 262, in clip_gradient
grad_norm = cupy.maximum(frobenius_norm(gradient), 1e-12)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vitaly/micromamba/envs/pc/lib/python3.11/site-packages/thinc/backends/cupy_ops.py", line 260, in frobenius_norm
return cupy.cublas.nrm2(X_vec)
^^^^^^^^^^^
File "/home/vitaly/micromamba/envs/pc/lib/python3.11/site-packages/cupy/__init__.py", line 927, in __getattr__
raise AttributeError(f"module 'cupy' has no attribute {name!r}")``
Fix
cupy.cublas.nrm2(X_vec) does not work, but from cupy import cublas; cublas.nrm2(X_vec) does. Sounds like an issue with cupy ...
The workaround is:
diff --git a/thinc/backends/cupy_ops.py b/thinc/backends/cupy_ops.py
index 1e1e5b92..dac4077a 100644
--- a/thinc/backends/cupy_ops.py
+++ b/thinc/backends/cupy_ops.py
@@ -257,7 +257,8 @@ class CupyOps(Ops):
# implementation.
def frobenius_norm(X):
X_vec = X.reshape(-1)
- return cupy.cublas.nrm2(X_vec)
+ from cupy import cublas
+ return cublas.nrm2(X_vec)
grad_norm = cupy.maximum(frobenius_norm(gradient), 1e-12)
gradient *= cupy.minimum(threshold, grad_norm) / grad_norm