I faked an RPC process in the way that you showed. The tuning result looked the same as before, but this time, the log has more information about the error. I’m investigating the stack trace now:
Trial #1: Error in running:
RPCRunner: An exception occurred
Traceback (most recent call last):
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/meta_schedule/runner/rpc_runner.py", line 377, in resource_handler
yield
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/meta_schedule/runner/rpc_runner.py", line 408, in _worker_func
repeated_args,
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/meta_schedule/runner/rpc_runner.py", line 515, in default_run_evaluator
return run_evaluator_common(rt_mod, device, evaluator_config, repeated_args)
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/meta_schedule/runner/utils.py", line 117, in run_evaluator_common
profile_result = evaluator(*args)
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/runtime/module.py", line 357, in evaluator
blob = feval(*args)
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
3: TVMFuncCall
2: tvm::runtime::RPCWrappedFunc::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
1: tvm::runtime::RPCClientSession::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)> const&)
0: tvm::runtime::RPCEndpoint::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)>)
File "/home/benjamin-gilby/tvm_env/tvm/src/runtime/rpc/rpc_endpoint.cc", line 804
TVMError:
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
Check failed: (code == RPCCode::kReturn) is false: code=kShutdown
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/exec/popen_worker.py", line 87, in main
result = fn(*args, **kwargs)
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/meta_schedule/runner/rpc_runner.py", line 408, in _worker_func
repeated_args,
File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
self.gen.throw(type, value, traceback)
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/meta_schedule/runner/rpc_runner.py", line 381, in resource_handler
f_cleanup(session, remote_path)
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/meta_schedule/runner/rpc_runner.py", line 532, in default_cleanup
session.remove(remote_path)
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/rpc/client.py", line 144, in remove
self._remote_funcs["remove"] = self.get_function("tvm.rpc.server.remove")
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/rpc/client.py", line 72, in get_function
return self._sess.get_function(name)
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/runtime/module.py", line 171, in get_function
self.handle, c_str(name), ctypes.c_int(query_imports), ctypes.byref(ret_handle)
File "/home/benjamin-gilby/tvm_env/tvm/python/tvm/_ffi/base.py", line 348, in check_call
raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
50: 0xffffffffffffffff
49: _start
48: __libc_start_main
47: _Py_UnixMain
46: 0x0000000000650da0
45: 0x0000000000650afa
44: _PyFunction_FastCallDict
43: _PyEval_EvalCodeWithName
42: _PyEval_EvalFrameDefault
41: _PyFunction_FastCallKeywords
40: _PyEval_EvalCodeWithName
39: _PyEval_EvalFrameDefault
38: _PyMethodDef_RawFastCallKeywords
37: 0x0000000000546369
36: _PyEval_EvalCodeWithName
35: _PyEval_EvalFrameDefault
34: _PyFunction_FastCallKeywords
33: _PyEval_EvalCodeWithName
32: _PyEval_EvalFrameDefault
31: _PyFunction_FastCallDict
30: _PyEval_EvalCodeWithName
29: _PyEval_EvalFrameDefault
28: _PyObject_FastCallDict
27: 0x00000000004c06e1
26: _PyFunction_FastCallDict
25: _PyEval_EvalFrameDefault
24: _PyMethodDescr_FastCallKeywords
23: 0x00000000005dcb58
22: 0x00000000005dc83f
21: 0x00000000004ba127
20: _PyEval_EvalFrameDefault
19: _PyFunction_FastCallKeywords
18: _PyEval_EvalFrameDefault
17: _PyFunction_FastCallKeywords
16: _PyEval_EvalFrameDefault
15: _PyFunction_FastCallKeywords
14: _PyEval_EvalFrameDefault
13: _PyFunction_FastCallKeywords
12: _PyEval_EvalCodeWithName
11: _PyEval_EvalFrameDefault
10: 0x0000000000537c30
9: _PyObject_FastCallKeywords
8: 0x00007fdd9f4fffa2
7: _ctypes_callproc
6: ffi_call
5: ffi_call_unix64
4: TVMModGetFunction
3: tvm::runtime::ModuleNode::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)
2: tvm::runtime::RPCModuleNode::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)
1: tvm::runtime::RPCClientSession::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
0: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::runtime::RPCEndpoint::Init()::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#2}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
File "/home/benjamin-gilby/tvm_env/tvm/src/runtime/rpc/rpc_endpoint.cc", line 684
TVMError:
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
Check failed: (code == RPCCode::kReturn) is false: code=1
