Hello,
I’m using TVM for almost 6 months, and I still have problems with reading backtraces. Example backtrace:
tvm._ffi.base.TVMError: Traceback (most recent call last):
4: TVMFuncCall
3: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<__mk_TVM1::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}> >::Call(tvm::runtime::PackedFuncObj const*, __mk_TVM1, tvm::runtime::TVMRetValue)
2: tvm::runtime::RPCDeviceAPI::GetAttr(DLDevice, tvm::runtime::DeviceAttrKind, tvm::runtime::TVMRetValue*)
1: non-virtual thunk to tvm::runtime::RPCClientSession::GetAttr(DLDevice, tvm::runtime::DeviceAttrKind, tvm::runtime::TVMRetValue*)
0: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::runtime::RPCEndpoint::Init()::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#2}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
File "/home/piotr/projects/odai-meta/odai-tvm/tvm/src/runtime/rpc/rpc_endpoint.cc", line 681
TVMError:
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
Check failed: (code == RPCCode::kReturn) is false: code=1
So, starting from the end. I need to go to src/runtime/rpc/rpc_endpoint.cc:681
. Then I see, thath ICHECK
is causing error. But ICHECK
isn’t causing error by itself, it is rather mechanism for checking errors.
So I need to go to line 0:
. First item is Extractor
as I can see in compte_inline.cc
, Extractor
is kind of wrapper/utility for loading buffers. I need to go deeper - I can see PackedFuncObj-like stuff. I think that in documentation it is TVM wrapper for many objects, so I can ignore it, and look at next elements. Next element in this puzzle is RPCEndpoint::Init
- looks like place where I can start looking for errors. Now I can see that ICHECK
was called in RPCEndpoint::Init
.
Nothing special in Init
- let’s look for 1:
, where the Init
is called? Thunks is explained in c++ - What is a 'thunk'? - Stack Overflow - so this is tiny function, which doesnt return value, but executes another function - can I go to 2:
to look for Init
call?
In 2:
I investigate RPCDeviceAPI::GetAttr
in src/runtime/rpc/rpc_device_api.cc:41
. 2 lines of code. Nothing special. No Init
call.
In 3:
the same constructions PackedFuncObject
of Extractor
of PackedFuncSubObject
. Lambda function, similar to 1:
. What is __mk_TVM1
? Looks like generated function, how can I see how it looks like? Is this function generater by tuner? What is this #1
? And what in 4:
?