[SOVLED] Failing runner RPC

Troubleshooting
#1

I’m getting folliwing error using RPC runner:

  File "/workspace/python/tvm/contrib/graph_executor.py", line 408, in benchmark
    func_name, device, repeat=repeat, number=number, min_repeat_ms=min_repeat_ms
  File "/workspace/python/tvm/runtime/module.py", line 292, in evaluator
    blob = feval(*args)
  File "/workspace/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
    raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
  4: TVMFuncCall
  3: _ZNSt17_Function_handlerIFvN3tvm7runtime7TVMArgsEPNS1_11
  2: tvm::runtime::RPCWrappedFunc::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
  1: tvm::runtime::RPCClientSession::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runt
ime::TVMArgs)> const&)
  0: tvm::runtime::RPCEndpoint::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::
TVMArgs)>)
  File "/workspace/src/runtime/rpc/rpc_endpoint.cc", line 801
TVMError:
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (code == RPCCode::kReturn) is false: code=kShutdown

Using another device with same parameters, dosnt’ cause problem

2 Likes
#2

The problem was timeout hardcoded in RPC Android apk, this value was TRACKER_TIMEOUT = 6000, and requested model when runing without tuning, was runing by average 16000ms.

#3

In similar setup, where I use tvm_rpc executable, problem was wrong compilation of tvm_rpc. I was thinking that I’m using OpenCL, but when I call in python logger.debug(f"dev.driver_version: {dev.driver_version}"), I’m getting following stacktrace:

Traceback (most recent call last):
  File "/usr/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/home/piotr/projects/odai-meta/odai-tvm/tvm/python/tvm/autotvm/measure/measure_methods.py", line 814, in _check
    logger.debug(f"dev.driver_version: {dev.driver_version}")
  File "/home/piotr/projects/odai-meta/odai-tvm/tvm/python/tvm/_ffi/runtime_ctypes.py", line 436, in driver_version
    return self._GetDeviceAttr(self.device_type, self.device_id, 12)
  File "/home/piotr/projects/odai-meta/odai-tvm/tvm/python/tvm/_ffi/runtime_ctypes.py", line 247, in _GetDeviceAttr
    return tvm.runtime._ffi_api.GetDeviceAttr(device_type, device_id, attr_id)
  File "/home/piotr/projects/odai-meta/odai-tvm/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
    raise get_last_ffi_error()
tvm.error.RPCError: Traceback (most recent call last):
  8: TVMFuncCall
  7: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<__mk_TVM1::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}> >::Call(tvm::runtime::PackedFuncObj const*, __mk_TVM1, tvm::runtime::TVMRetValue)
  6: tvm::runtime::RPCDeviceAPI::GetAttr(DLDevice, tvm::runtime::DeviceAttrKind, tvm::runtime::TVMRetValue*)
  5: non-virtual thunk to tvm::runtime::RPCClientSession::GetAttr(DLDevice, tvm::runtime::DeviceAttrKind, tvm::runtime::TVMRetValue*)
  4: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::runtime::RPCEndpoint::Init()::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#2}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
  3: tvm::runtime::RPCEndpoint::HandleUntilReturnEvent(bool, std::function<void (tvm::runtime::TVMArgs)>)
  2: tvm::runtime::RPCEndpoint::EventHandler::HandleNextEvent(bool, bool, std::function<void (tvm::runtime::TVMArgs)>)
  1: tvm::runtime::RPCEndpoint::EventHandler::HandleProcessPacket(std::function<void (tvm::runtime::TVMArgs)>)
  0: tvm::runtime::RPCEndpoint::EventHandler::HandleReturn(tvm::runtime::RPCCode, std::function<void (tvm::runtime::TVMArgs)>)
  File "/home/piotr/projects/odai-meta/odai-tvm/tvm/src/runtime/rpc/rpc_endpoint.cc", line 376
RPCError: Error caught from RPC call:
[12:58:36] /workspace/src/runtime/c_runtime_api.cc:131: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (allow_missing) is false: Device API opencl is not enabled.
#4

Hi, I have find parameter TRACKER_TIMEOUT in file jvm/core/src/main/java/org/apache/tvm/rpc/ConnectTrackerServerProcessor.java. I want to know that how to make this parameter take effect. I only rebuild the android_rpc but it seems takes no effect.

#5

Not sure about your setup, and my previous setup(sic!). But I think java files are for android application. If you think android_rpc is binary executable which can be run in shell, then you need rebuild with -DUSE_CPP_RPC=ON.