Segfault while importing TVM

Troubleshooting
#1

Segmentation fault (core dumped) when running python -c "import tvm" after compiling TVM successfully follow this doc.

I’ve tried using faulthandler to locate the error:

#code.py
import faulthandler
faulthandler.enable()
import tvm

After running python -X faulthandler code.py, I got messages below:

Fatal Python error: Segmentation fault

Current thread 0x00007f01d5705280 (most recent call first):
  File "/root/tvm-18/python/tvm/_ffi/_ctypes/packed_func.py", line 235 in __call__
  File "/root/tvm-18/python/tvm/target/tag.py", line 33 in list_tags
  File "/root/tvm-18/python/tvm/target/tag.py", line 81 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 843 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "/root/tvm-18/python/tvm/target/__init__.py", line 77 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 843 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1042 in _handle_fromlist
  File "/root/tvm-18/python/tvm/__init__.py", line 58 in <module>
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 843 in exec_module
  File "<frozen importlib._bootstrap>", line 671 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 975 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 991 in _find_and_load
  File "code.py", line 3 in <module>
Segmentation fault (core dumped)

There seems to be something wrong with the fuction in tvm/python/tvm/_ffi/_ctypes/packed_func.py as mentioned above while executing in my Environment.

Environment

  • Ubuntu 20.04.6 LTS
  • Docker image of 12.4.1-cudnn-devel-ubuntu20.04
  • LLVM-18.1.8 prebuild version for Ubuntu 18.04: (download from here)
  • Use Apache TVM v0.18.0 Release version, and config.cmake appending settings are: set(USE_CUDA ON), set(USE_CUBLAS ON), set(USE_CUDNN ON), set(USE_LLVM "\<path-to-llvm-config\> --ignore-libllvm --link-static") and set(HIDE_PRIVATE_SYMBOLS ON)
  • Correctly set TVM_HOME and PYTHONPATH, use conda venv and python 3.8.
#2

New discovery: This issue may be caused by a version of LLVM that is too high. When the LLVM version used is 16 or above, it will compile code that leads to a Segmentation Fault, but the compilation itself passes; however, when using LLVM 15, everything works as expected. Magic…