When launching any graph I built, I get this stack trace from TVM:
> ???
E tvm._ffi.base.TVMError: Traceback (most recent call last):
E [bt] (8) /home/jenkins/miniconda/lib/libtvm.so(+0x9f5f69) [0x7f0bb0435f69]
E [bt] (7) /home/jenkins/miniconda/lib/libtvm.so(+0x9f2fde) [0x7f0bb0432fde]
E [bt] (6) /home/jenkins/miniconda/lib/libtvm.so(+0x9f2134) [0x7f0bb0432134]
E [bt] (5) /home/jenkins/miniconda/lib/libtvm.so(+0x9e4ed4) [0x7f0bb0424ed4]
E [bt] (4) /home/jenkins/miniconda/lib/libtvm.so(+0x9f6ed1) [0x7f0bb0436ed1]
E [bt] (3) /home/jenkins/miniconda/lib/libtvm.so(+0x9f5726) [0x7f0bb0435726]
E [bt] (2) /home/jenkins/miniconda/lib/libtvm.so(+0x9eb984) [0x7f0bb042b984]
E [bt] (1) /home/jenkins/miniconda/lib/libtvm.so(+0x9c1056) [0x7f0bb0401056]
E [bt] (0) /home/jenkins/miniconda/lib/libtvm.so(+0xe4655c) [0x7f0bb088655c]
E File "tvm/_ffi/_cython/./function.pxi", line 56, in tvm._ffi._cy3.core.tvm_callback
E File "/home/jenkins/miniconda/lib/python3.7/site-packages/tvm/relay/backend/_backend.py", line 89, in build
E return _build.build(funcs, target=target, target_host=target_host)
E File "/home/jenkins/miniconda/lib/python3.7/site-packages/tvm/build_module.py", line 621, in build
E fhost, mdev = _build_for_device(flist, tar, target_host)
E File "/home/jenkins/miniconda/lib/python3.7/site-packages/tvm/build_module.py", line 488, in _build_for_device
E mdev = codegen.build_module(fdevice, str(target)) if fdevice else None
E File "/home/jenkins/miniconda/lib/python3.7/site-packages/tvm/codegen.py", line 36, in build_module
E return _Build(lowered_func, target)
E File "tvm/_ffi/_cython/./function.pxi", line 310, in tvm._ffi._cy3.core.FunctionBase.__call__
E File "tvm/_ffi/_cython/./function.pxi", line 245, in tvm._ffi._cy3.core.FuncCall
E File "tvm/_ffi/_cython/./function.pxi", line 234, in tvm._ffi._cy3.core.FuncCall3
E File "tvm/_ffi/_cython/./base.pxi", line 170, in tvm._ffi._cy3.core.CALL
E [bt] (5) /home/jenkins/miniconda/lib/libtvm.so(TVMFuncCall+0x65) [0x7f0bb0886f85]
E [bt] (4) /home/jenkins/miniconda/lib/libtvm.so(+0x675ec3) [0x7f0bb00b5ec3]
E [bt] (3) /home/jenkins/miniconda/lib/libtvm.so(+0x768408) [0x7f0bb01a8408]
E [bt] (2) /home/jenkins/miniconda/lib/libtvm.so(+0x795d04) [0x7f0bb01d5d04]
E [bt] (1) /home/jenkins/miniconda/lib/libtvm.so(+0xdebe1e) [0x7f0bb082be1e]
E [bt] (0) /home/jenkins/miniconda/lib/libtvm.so(+0xe4655c) [0x7f0bb088655c]
E File "tvm/_ffi/_cython/./function.pxi", line 56, in tvm._ffi._cy3.core.tvm_callback
E File "/home/jenkins/miniconda/lib/python3.7/site-packages/tvm/autotvm/measure/measure_methods.py", line 574, in tvm_callback_cuda_compile
E ptx = nvcc.compile_cuda(code, target="ptx", arch=AutotvmGlobalScope.current.cuda_target_arch)
E File "/home/jenkins/miniconda/lib/python3.7/site-packages/tvm/contrib/nvcc.py", line 91, in compile_cuda
E cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
E File "/home/jenkins/miniconda/lib/python3.7/subprocess.py", line 775, in __init__
E restore_signals, start_new_session)
E File "/home/jenkins/miniconda/lib/python3.7/subprocess.py", line 1522, in _execute_child
E raise child_exception_type(errno_num, err_msg, err_filename)
E FileNotFoundError: [Errno 2] No such file or directory: 'nvcc': 'nvcc'
This seems to come from autotvm, more specifically, this method:
tvm/autotvm/measure/measure_methods.py, line 574, in tvm_callback_cuda_compile
I’m trying to avoid users having to install CUDA themselves and nvcc is not redistributable.
Is there a way to use NVRTC instead? I can have some code to set hooks if required.