GDB attach debug TVM

Troubleshooting
#1

I was hoping you could assist me with a couple of questions regarding debugging the quantization part of TVM using a Python script and GDB attach. Specifically, I have the following questions:

  1. Does the quantization process call functions in src/relay/quantize?
  2. Since I am uncertain which part of TVM the quantization process calls, I would like to debug a known issue in order to test whether GDB attach is working properly.

One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.
[18:24:11] /home/ubuntu/mxnet/src/io/iter_image_recordio_2.cc:180: ImageRecordIOParser2: /home/ubuntu/.tvm_test_data/val_256_q90.rec, use 4 threads for decoding..
[18:24:11] /home/ubuntu/mxnet/src/storage/storage.cc:202: Using Pooled (Naive) StorageManager for CPU_PINNED
Traceback (most recent call last):
  File "/home/ubuntu/scripts/compile_quant_pytorch.py", line 160, in <module>
    quant_tvm_model = graph_executor.GraphModule(lib["default"](dev))
  File "/home/ubuntu/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
    raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
  5: TVMFuncCall
  4: tvm::runtime::PackedFuncObj::Extractor<tvm::runtime::PackedFuncSubObj<tvm::runtime::GraphExecutorFactory::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}> >::Call(tvm::runtime::PackedFuncObj const*, tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)
  3: tvm::runtime::GraphExecutorFactory::ExecutorCreate(std::vector<DLDevice, std::allocator<DLDevice> > const&)
  2: tvm::runtime::GraphExecutor::Init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tvm::runtime::Module, std::vector<DLDevice, std::allocator<DLDevice> > const&, tvm::runtime::PackedFunc)
  1: tvm::runtime::GraphExecutor::SetupOpExecs()
  0: tvm::runtime::GraphExecutor::CreateTVMOp(tvm::runtime::TVMOpParam const&, std::vector<DLTensor, std::allocator<DLTensor> > const&)
  File "/home/ubuntu/tvm/src/runtime/graph_executor/graph_executor.cc", line 585
TVMError: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (pf != nullptr) is false: no such function in module: quantize_inputs

I set a breakpoint on line 585 in the file /home/ubuntu/tvm/src/runtime/graph_executor/graph_executor.cc. However, it seems that the code does not pause at that point and instead raises get_last_ffi_error(). Is function actually called in the Python script graph_executor.GraphModule?

Thank you in advance for your help!

#2

Are you using a debug build of TVM? That is, the CMAKE_BUILD_TYPE has to be set to Debug for gdb to work properly with TVM…

#3

Yes, need to build with -DCMAKE_BUILD_TYPE=Debug. Thanks so much for your help! :sob: