Hi, I used TVM 0.7.0 to compile my ONNX model to TVM model to run on Jetson Nano GPU. The compilation was successful, but the running had some issues which are quite similar to 0.7.dev1 cudaErrorCudartUnloading.
Traceback (most recent call last): [122/1881]
File "latency_tvm.py", line 80, in <module>
prof_res = np.array(ftimer().results) * 1e3
File "/home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/runtime/module.py", line 226, in evaluator
blob = feval(*args)
File "tvm/_ffi/_cython/./packed_func.pxi", line 322, in tvm._ffi._cy3.core.PackedFuncBase.__call__
File "tvm/_ffi/_cython/./packed_func.pxi", line 257, in tvm._ffi._cy3.core.FuncCall
File "tvm/_ffi/_cython/./packed_func.pxi", line 246, in tvm._ffi._cy3.core.FuncCall3
File "tvm/_ffi/_cython/./base.pxi", line 160, in tvm._ffi._cy3.core.CALL
tvm._ffi.base.TVMError: Traceback (most recent call last):
[bt] (4) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(TVMFuncCall+0x70) [0x7f361336f0]
[bt] (3) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(+0x10ced54) [0x7f3619cd54]
[bt] (2) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(+0x10cea9c) [0x7f3619ca9c]
[bt] (1) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(tvm::runtime::CUDADeviceAPI::StreamSync(DLContext, void*)+0x13c) [0x7f361c8674]
[bt] (0) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(+0x10f819c) [0x7f361c619c]
File "/home/nano/incubator-tvm/src/runtime/cuda/cuda_device_api.cc", line 195
CUDA: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: the launch timed out and was terminated
Traceback (most recent call last):
File "tvm/_ffi/_cython/./base.pxi", line 160, in tvm._ffi._cy3.core.CALL
tvm._ffi.base.TVMError: Traceback (most recent call last):
[bt] (3) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(TVMArrayFree+0x48) [0x7f36159300]
[bt] (2) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(tvm::runtime::NDArray::Internal::DefaultDeleter(tvm::runtime::Object*)+0x78) [0x
7f3615c740]
[bt] (1) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(tvm::runtime::CUDADeviceAPI::FreeDataSpace(DLContext, void*)+0x1d4) [0x7f361c819
4]
[bt] (0) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(+0x10f819c) [0x7f361c619c]
File "/home/nano/incubator-tvm/src/runtime/cuda/cuda_device_api.cc", line 126
CUDA: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: the launch timed out and was terminated
Exception ignored in: 'tvm._ffi._cy3.core.NDArrayBase.__dealloc__'
Traceback (most recent call last):
File "tvm/_ffi/_cython/./base.pxi", line 160, in tvm._ffi._cy3.core.CALL
tvm._ffi.base.TVMError: Traceback (most recent call last):
[bt] (3) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(TVMArrayFree+0x48) [0x7f36159300]
[bt] (2) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(tvm::runtime::NDArray::Internal::DefaultDeleter(tvm::runtime::Object*)+0x78) [0x
7f3615c740]
[bt] (1) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(tvm::runtime::CUDADeviceAPI::FreeDataSpace(DLContext, void*)+0x1d4) [0x7f361c819
4]
[bt] (0) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(+0x10f819c) [0x7f361c619c]
File "/home/nano/incubator-tvm/src/runtime/cuda/cuda_device_api.cc", line 126
CUDA: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: the launch timed out and was terminated
terminate called after throwing an instance of 'dmlc::Error'
what(): [12:45:14] /home/nano/incubator-tvm/src/runtime/cuda/cuda_device_api.cc:126: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: CUDA: the launch timed out and w
as terminated
Stack trace:
[bt] (0) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(+0x10f819c) [0x7f361c619c]
[bt] (1) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(tvm::runtime::CUDADeviceAPI::FreeDataSpace(DLContext, void*)+0x1d4) [0x7f361c819
4]
[bt] (2) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(tvm::runtime::NDArray::Internal::DefaultDeleter(tvm::runtime::Object*)+0x78) [0x
7f3615c740]
[bt] (3) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(tvm::runtime::GraphRuntime::~GraphRuntime()+0x474) [0x7f361ba154]
[bt] (4) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(tvm::runtime::SimpleObjAllocator::Handler<tvm::runtime::GraphRuntime>::Deleter_(
tvm::runtime::Object*)+0x20) [0x7f361ba1b0]
[bt] (5) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(+0x10e57d0) [0x7f361b37d0]
[bt] (6) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(+0x10cefb0) [0x7f3619cfb0]
[bt] (7) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/libtvm.so(TVMFuncFree+0x28) [0x7f36130590]
[bt] (8) /home/nano/.local/lib/python3.6/site-packages/tvm-0.7.0-py3.6-linux-aarch64.egg/tvm/_ffi/_cy3/core.cpython-36m-aarch64-linux-gnu.so(+0xfcc0) [0x7f2f792cc0]
I’ve tried the solution in 0.7.dev1 cudaErrorCudartUnloading but it seemed it did not work. If my compilation target is cpu, the compiled model could run normally on the CPU of Jetson Nano. I wonder how I could fix this running issue on Jetson Nano GPU?