How to debug TVMRuntime

lyq · April 23, 2018, 10:24am

Hi everyone

I want to inspect TVMRuntime. I have modified config.mk to “DEBUG = 1”.

Then try to debug in gdb by running a python script

I have set breakpoints in c_runtime_api.cc and graph_runtime.cc.But it didn’t stop at all.

Any pieces of advice?

Thanked in advance！

nhynes · April 23, 2018, 10:49pm

What happens if you try

gdb python
(gdb) break <my_breakpoints>
(gdb) run run.py

xqdan · April 24, 2018, 12:56am

You are trying to debug a python script with c++ code behind.
This may be helpful,
https://wiki.python.org/moin/DebuggingWithGdb

lyq · April 24, 2018, 1:12am

Thanks for your reply.

In the third picture, I have set breakpoints in C++ source file, but it didn’t stop at all.

nhynes · April 24, 2018, 1:26am

maybe compile with ADD_CFLAGS="-g -Og" and remove any -O2s you find?

blueyi · September 29, 2018, 8:55am

If you like IDE, recommend you try visual studio 2017, but you have to compile tvm in windows with msbuild, and then just create python project, now you can step into tvm runtime lib, just like you debug a normal c++ library

nhynes · September 30, 2018, 5:30am

@blueyi is there a way to do this that doesn’t require VS? Perhaps gdb or something?

blueyi · September 30, 2018, 6:01am

Another recommend is same as @xqdan

wweic · September 30, 2018, 8:05am

In my experience, starting python under GDB usually causes all kinds of arcane issues. It’s much better to attach GDB to a running python program to avoid that. For example, your issue seems to be that you are setting breakpoints on non-existent address, I’m guessing that you set your breakpoints before tvm related shared libraries are loaded. GDB may or may not fix the symbol address for you.

I would recommend the following extra simple steps:

Add input("dummy breakpoint") after the last import statement(from tvm.contrib import graph_runtime). So the program will stop after loading all tvm shared libraries.
In a separate window, find the process id of the python program and attach GDB to the process, set the breakpoints you want and continue the program so GDB won’t pause the python process.
Go back to python program and type anything just to resume the python program and it will hit your breakpoint!

Just verified the steps with lldb:

➜  lldb --attach-pid 42981
(lldb) process attach --pid 42981
Process 42981 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff5c8eaec2 libsystem_kernel.dylib`__read_nocancel + 10
libsystem_kernel.dylib`__read_nocancel:

(lldb) breakpoint set --name NNGraphCreate
Breakpoint 1: where = libnnvm_compiler.dylib`NNGraphCreate, address = 0x000000011a130c60
(lldb) c
Process 42981 resuming
Process 42981 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x000000011a130c60 libnnvm_compiler.dylib`NNGraphCreate
libnnvm_compiler.dylib`NNGraphCreate:
->  0x11a130c60 <+0>: pushq  %rbp
    0x11a130c61 <+1>: movq   %rsp, %rbp
    0x11a130c64 <+4>: pushq  %r15
    0x11a130c66 <+6>: pushq  %r14
Target 0: (Python) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x000000011a130c60 libnnvm_compiler.dylib`NNGraphCreate
    frame #1: 0x000000010a50e3ef _ctypes.cpython-37m-darwin.so`ffi_call_unix64 + 79
    frame #2: 0x000000010a50eb93 _ctypes.cpython-37m-darwin.so`ffi_call + 703
    frame #3: 0x000000010a50a4ab _ctypes.cpython-37m-darwin.so`_ctypes_callproc + 662
    frame #4: 0x000000010a504f98 _ctypes.cpython-37m-darwin.so`PyCFuncPtr_call + 520
    frame #5: 0x0000000109bbd315 Python`_PyObject_FastCallKeywords + 359
    frame #6: 0x0000000109c54173 Python`call_function + 568
    frame #7: 0x0000000109c4b305 Python`_PyEval_EvalFrameDefault + 2604
    frame #8: 0x0000000109bbd88e Python`function_code_fastcall + 116
    frame #9: 0x0000000109c5417a Python`call_function + 575
    frame #10: 0x0000000109c4b305 Python`_PyEval_EvalFrameDefault + 2604
    frame #11: 0x0000000109c54ad1 Python`_PyEval_EvalCodeWithName + 1837
    frame #12: 0x0000000109c4a851 Python`PyEval_EvalCode + 42
    frame #13: 0x0000000109c79a5f Python`run_mod + 54
    frame #14: 0x0000000109c78a7a Python`PyRun_FileExFlags + 164
    frame #15: 0x0000000109c78159 Python`PyRun_SimpleFileExFlags + 283
    frame #16: 0x0000000109c8f842 Python`pymain_main + 5114
    frame #17: 0x0000000109c8ffd4 Python`_Py_UnixMain + 104
    frame #18: 0x00007fff5c79b115 libdyld.dylib`start + 1
    frame #19: 0x00007fff5c79b115 libdyld.dylib`start + 1

johnmattew490 · January 16, 2020, 5:16pm

I was wondering is it has to be MS 2017 ? Is there a Linux based IDE such as CLION to debug TVM?

johnmattew490 · January 17, 2020, 1:48am

Hello blueyi,

Could you please give little bit help?

I have build TVM with DEBUG, and I want to step into relay.build_module.build() python function and (my goal) is to step into underlying C++ code form python.

My python project is in PyCharm, and I am lost from here. My goal is to start from python (PyCharm environment) and hit any debug point in C++ code which might be the entry for the relay.build_module.build() python function.

apivovarov · January 17, 2020, 8:14pm

I have real task to debug Runtime failed for 5D input