Can't compile for CUDA GPU

1

I had a working system last week, then I went away. When I came back I haven’t been able to compile anything for cuda. I get this error using the code from the from_keras.py tutorial.

Here is the full error:

Traceback (most recent call last):
  File "original_from_keras.py", line 119, in <module>
    main()
  File "original_from_keras.py", line 80, in main
    graph, lib, params = nnvm.compiler.build(sym, target, shape_dict, params=params)
  File "/home/jimmy/projects/autotvm/nnvm/python/nnvm/compiler/build_module.py", line 304, in build
    graph = graph.apply("GraphCompile")
  File "/home/jimmy/projects/autotvm/nnvm/python/nnvm/graph.py", line 234, in apply
    check_call(_LIB.NNGraphApplyPasses(self.handle, npass, cpass, ctypes.byref(ghandle)))
  File "/home/jimmy/projects/autotvm/nnvm/python/nnvm/_base.py", line 75, in check_call
    raise NNVMError(py_str(_LIB.NNGetLastError()))
nnvm._base.NNVMError: TVMCall CFunc Error:
Traceback (most recent call last):
  File "/home/jimmy/projects/autotvm/python/tvm/_ffi/_ctypes/function.py", line 54, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/jimmy/projects/autotvm/nnvm/python/nnvm/compiler/build_module.py", line 124, in _build
    return tvm.build(funcs, target=target, target_host=target_host)
  File "/home/jimmy/projects/autotvm/python/tvm/build_module.py", line 504, in build
    mdev = codegen.build_module(fdevice, str(target_device))
  File "/home/jimmy/projects/autotvm/python/tvm/codegen.py", line 20, in build_module
    return _Build(lowered_func, target)
  File "/home/jimmy/projects/autotvm/python/tvm/_ffi/function.py", line 280, in my_api_func
    return flocal(*args)
  File "/home/jimmy/projects/autotvm/python/tvm/_ffi/_ctypes/function.py", line 184, in __call__
    ctypes.byref(ret_val), ctypes.byref(ret_tcode)))
  File "/home/jimmy/projects/autotvm/python/tvm/_ffi/base.py", line 66, in check_call
    raise TVMError(py_str(_LIB.TVMGetLastError()))
TVMError: TVMCall CFunc Error:
Traceback (most recent call last):
  File "/home/jimmy/projects/autotvm/python/tvm/_ffi/_ctypes/function.py", line 54, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/jimmy/projects/autotvm/python/tvm/autotvm/measure/measure_methods.py", line 487, in tvm_callback_cuda_compile
    ptx = nvcc.compile_cuda(code, target="ptx", arch=AutotvmGlobalScope.current.cuda_target_arch)
  File "/home/jimmy/projects/autotvm/python/tvm/contrib/nvcc.py", line 69, in compile_cuda
    cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I’ve tried removing and reinstalling tensorflow and keras, then pulling the latest commit of tvm and creating a fresh build. I still get the same error. if I change the code to run on a cpu using llvm it works fine.

2

As a sanity check, do the CUDA samples that came with your CUDA install e.g., deviceQuery build and run?

After verifying that I would try to bisect which recent commit broke the script.

3

Have you verified that your CUDA environment is set up correctly (CUDA_ROOT, PATH, LD_LIBRARY_PATH)? For example, the system I’m working uses a modules environment, and if I forget to load the CUDA module I get a very similar error when running from_keras.py.

4

I had already checked the CUDA samples and they were running fine.

I just took a look at my environment and added cuda to my PATH, this seems to have solved the problem.

I’m not sure what changed in the system, because everything else was working without this, and so was TVM until recently.

Thanks for the replies!