Reading TVM tracebacks

Hello,

I’m using TVM for almost 6 months, and I still have problems with reading backtraces. Example backtrace:

tvm._ffi.base.TVMError: Traceback (most recent call last):
  4: TVMFuncCall
  3: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<__mk_TVM1::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}> >::Call(tvm::runtime::PackedFuncObj const*, __mk_TVM1, tvm::runtime::TVMRetValue)
  2: tvm::runtime::RPCDeviceAPI::GetAttr(DLDevice, tvm::runtime::DeviceAttrKind, tvm::runtime::TVMRetValue*)
  1: non-virtual thunk to tvm::runtime::RPCClientSession::GetAttr(DLDevice, tvm::runtime::DeviceAttrKind, tvm::runtime::TVMRetValue*)
  0: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::runtime::RPCEndpoint::Init()::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#2}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
  File "/home/piotr/projects/odai-meta/odai-tvm/tvm/src/runtime/rpc/rpc_endpoint.cc", line 681
TVMError: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (code == RPCCode::kReturn) is false: code=1

So, starting from the end. I need to go to src/runtime/rpc/rpc_endpoint.cc:681. Then I see, thath ICHECK is causing error. But ICHECK isn’t causing error by itself, it is rather mechanism for checking errors.

So I need to go to line 0:. First item is Extractor as I can see in compte_inline.cc, Extractor is kind of wrapper/utility for loading buffers. I need to go deeper - I can see PackedFuncObj-like stuff. I think that in documentation it is TVM wrapper for many objects, so I can ignore it, and look at next elements. Next element in this puzzle is RPCEndpoint::Init - looks like place where I can start looking for errors. Now I can see that ICHECK was called in RPCEndpoint::Init.

Nothing special in Init - let’s look for 1:, where the Init is called? Thunks is explained in c++ - What is a 'thunk'? - Stack Overflow - so this is tiny function, which doesnt return value, but executes another function - can I go to 2: to look for Init call?

In 2: I investigate RPCDeviceAPI::GetAttr in src/runtime/rpc/rpc_device_api.cc:41. 2 lines of code. Nothing special. No Init call.

In 3: the same constructions PackedFuncObject of Extractor of PackedFuncSubObject. Lambda function, similar to 1:. What is __mk_TVM1? Looks like generated function, how can I see how it looks like? Is this function generater by tuner? What is this #1? And what in 4:?