vinx13
1
When USE_THRUST=ON
, unknown CUDA error happened:
File "/home/ubuntu/tvm/src/runtime/cuda/cuda_device_api.cc", line 108
CUDA: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: unknown error
It can be reproduced with the following script
import numpy as np
import tvm
import xgboost
if __name__=='__main__':
a_np = np.ones((1, 16))
a_tvm = tvm.nd.array(a_np, ctx=tvm.gpu(0))
My environment:
CUDA 10.0, thrust 1.9.3
XGBoost: 1.1.0 (installed with pip or from source)
I found a workaround is to import xgboost before importing TVM
@hcho3 looks like another symbol conflict?
no it doesn’t look like that…
hcho3
3
@vinx13 I cannot reproduce the issue on my machine. Can you reproduce the issue in a Docker container?
vinx13
4
Unfortunately I’m not able to reproduce in a docker right now. I’ll update here if I find a way to reproduce it
Met same problem when open USE_THRUST=ON
without using xgboost
My environment: CUDA 10.1, thrust 1.9.5
I got the same problem and filed a PR to workaround it.