I am writing a function which uses and topk, it works well on CPU but can fail occasionally with the following error, it doesn’t happen every time, but when it happens, it always happens after another CUDA call, this error will not be triggered if it is called and tested by it self.
> raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
[bt] (3) /home/developer/tvm/build/libtvm.so(TVMArrayCopyToBytes+0xa) [0x7fec95ca9f6a]
[bt] (2) /home/developer/tvm/build/libtvm.so(tvm::runtime::ArrayCopyToBytes(DLTensor const*, void*, unsigned long)+0x250) [0x7fec95ca9e00]
[bt] (1) /home/developer/tvm/build/libtvm.so(tvm::runtime::CUDADeviceAPI::CopyDataFromTo(void const*, unsigned long, void*, unsigned long, unsigned long, DLContext, DLContext, DLDataType, void*)+0x99) [0x7fec95d22419]
[bt] (0) /home/developer/tvm/build/libtvm.so(+0x12d80b2) [0x7fec95d1f0b2]
File "/home/developer/tvm/src/runtime/cuda/cuda_device_api.cc", line 223
TVMError:
---------------------------------------------------------------
An internal invariant was violated during the execution of TVM.
Please read TVM's error reporting guidelines.
More details can be found here: https://discuss.tvm.ai/t/error-reporting/7793.
---------------------------------------------------------------
Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading == false: CUDA: an illegal memory access was encountered
There are bound errors during the runtime, I am not sure if there is something related.
/home/developer/tvm/src/te/schedule/bound.cc:119: not in feed graph consumer = extern(argsort_gpu, 0x55fae219d260)
/home/developer/tvm/src/te/schedule/bound.cc:119: not in feed graph consumer = extern(topk_gpu, 0x55fae3ba2a50)
/home/developer/tvm/src/te/schedule/bound.cc:119: not in feed graph consumer = extern(topk_gpu, 0x55fae3ba2a50)
/home/developer/tvm/src/te/schedule/bound.cc:119: not in feed graph consumer = extern(topk_gpu, 0x55fae3ba2a50)
So I wonder if there is anyone who has encountered a similar issue? and what could potentially be the root of the issue?