This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
Conv1D throws CUDNN_STATUS_EXECUTION_FAILED #11241
Description
Setup:
$ pip install mxnet-cu90==1.1.0
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
$ ls /usr/local/cuda/lib64/libcudnn.so.7.0.3
/usr/local/cuda/lib64/libcudnn.so.7.0.3
Run the following script debug.py:
import mxnet as mx
W_REQ = 'add'
shape = (1, 65536, 1)
ctx = mx.gpu()
kwargs = {'no_bias': True, 'kernel': (1,), 'num_filter': 1}
x = mx.sym.var('x')
w = mx.sym.var('w')
x_grad = mx.nd.zeros(shape, ctx=ctx)
w_grad = mx.nd.zeros(shape, ctx=ctx)
args_grad = {'x': x_grad, 'w': w_grad}
sym = mx.sym.Convolution(x, w, **kwargs)
executor = sym.bind(ctx, grad_req={'x': 'null', 'w': W_REQ},
args={'x': mx.nd.ones(shape, ctx=ctx), 'w': mx.nd.ones(shape, ctx=ctx)},
args_grad=args_grad)
executor.forward()
executor.backward([mx.nd.ones((1,1,1), ctx=ctx)])
mx.nd.waitall()
Gives the following error:
[06:31:41] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
terminate called after throwing an instance of 'dmlc::Error'
what(): [06:31:41] src/engine/./threaded_engine.h:359: [06:31:41] src/operator/nn/./cudnn/cudnn_convolution-inl.h:242: Check failed: e == CUDNN_STATUS_SUCCESS (8 vs. 0) cuDNN: CUDNN_STATUS_EXECUTION_FAILED
Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a9e78) [0x7f705d9a3e78]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2aa288) [0x7f705d9a4288]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a920d1) [0x7f706018c0d1]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x262f5e7) [0x7f705fd295e7]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x24570bb) [0x7f705fb510bb]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x245d7d4) [0x7f705fb577d4]
[bt] (6) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243e2ed) [0x7f705fb382ed]
[bt] (7) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442bdb) [0x7f705fb3cbdb]
[bt] (8) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442db6) [0x7f705fb3cdb6]
[bt] (9) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243f68b) [0x7f705fb3968b]
A fatal error occurred in asynchronous engine operation. If you do not know what caused this error, you can try set environment variable MXNET_ENGINE_TYPE to NaiveEngine and run with debugger (i.e. gdb). This will force all operations to be synchronous and backtrace will give you the series of calls that lead to this error. Remember to set MXNET_ENGINE_TYPE back to empty after debugging.
Stack trace returned 9 entries:
[bt] (0) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2a9e78) [0x7f705d9a3e78]
[bt] (1) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2aa288) [0x7f705d9a4288]
[bt] (2) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243e594) [0x7f705fb38594]
[bt] (3) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442bdb) [0x7f705fb3cbdb]
[bt] (4) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x2442db6) [0x7f705fb3cdb6]
[bt] (5) /usr/local/lib/python2.7/dist-packages/mxnet/libmxnet.so(+0x243f68b) [0x7f705fb3968b]
[bt] (6) /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80) [0x7f70d400bc80]
[bt] (7) /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f70d52a66ba]
[bt] (8) /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f70d4fdc3dd]
Note that there's no error if W_REQ is changed to 'write'.
Can also be reproduced if I build mxnet from source at commit 5b99b25 where Conv1D CUDNN was initially introduced.