TVM build segfaults on the provided microtvm vagrant VM

Troubleshooting
#1

I have followed GitHub - areusch/microtvm-blogpost-eval to setup a microTVM vagrant VM (using the Vagrantfile in tvm/apps/microtvm/reference-vm/zephyr). Because it is a reference VM, I assumed it would not have any problem building TVM. However, when building TVM inside the vagrant VM, I get segfault. Since I am just using the baseline VM that is provided by the team, I am not sure why this would fail.

Are there anything I should be aware of when using the baseline VM?

Attaching my error log. Thank you

=======BELOW IS THE ERROR LOG=========

vagrant@microtvm:/mnt/d/tvm/apps/microtvm/reference-vm/zephyr$ ./setup.sh
+ cd ../../../../
+ apps/microtvm/reference-vm/zephyr/rebuild-tvm.sh
-- Build with RPC support...
-- Build with Graph Executor support...
-- Build with profiler...
-- VTA build with VTA_HW_PATH=/mnt/d/tvm/3rdparty/vta-hw
-- Build VTA runtime with target: sim
-- Build standalone CRT for micro TVM
-- Link with dynamic LLVM library
-- Found LLVM_INCLUDE_DIRS=/usr/lib/llvm-6.0/include
-- Found LLVM_DEFINITIONS=-D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
-- Found LLVM_LIBS=LLVM
-- Found TVM_LLVM_VERSION=60
-- Build with LLVM 6.0.0
-- Set TVM_LLVM_VERSION=60
-- Build with Micro support
-- Build with contrib.random
-- Build with contrib.sort
-- Build with contrib.hybriddump
-- Git found: /usr/bin/git
-- Found TVM_GIT_COMMIT_HASH=81afb14c4e532db1631cb005ba1f9c0ac62546e7
-- Autoset: USE_LIBBACKTRACE=ON in Linux
-- Building with libbacktrace...
-- Building with TVM Map...
-- Build with thread support...
-- Configuring done
-- Generating done
-- Build files have been written to: /mnt/d/tvm/build-microtvm
make: Warning: File 'Makefile' has modification time 0.053 s in the future
[  0%] Generating standalone_crt/src/runtime/crt/utvm_rpc_common/crcccitt.c
[  0%] Generating standalone_crt/include/checksum.h
[  0%] Generating standalone_crt/src/runtime/crt/tab/gentab_ccitt.inc
[  0%] Generating standalone_crt/include/dlpack/dlpack.h
[  2%] Built target project_libbacktrace
[  2%] Generating standalone_crt/include/dmlc/any.h
[  3%] Generating standalone_crt/include/dmlc/array_view.h
[  3%] Generating standalone_crt/include/dmlc/base.h
[  3%] Generating standalone_crt/include/dmlc/blockingconcurrentqueue.h
[  3%] Generating standalone_crt/include/dmlc/build_config_default.h
[  3%] Generating standalone_crt/include/dmlc/common.h
[  4%] Generating standalone_crt/include/dmlc/concurrency.h
[  4%] Generating standalone_crt/include/dmlc/concurrentqueue.h
[  4%] Generating standalone_crt/include/dmlc/config.h
[  4%] Generating standalone_crt/include/dmlc/data.h
[  4%] Generating standalone_crt/include/dmlc/endian.h
[  4%] Generating standalone_crt/include/dmlc/filesystem.h
[  5%] Generating standalone_crt/include/dmlc/input_split_shuffle.h
[  5%] Generating standalone_crt/include/dmlc/io.h
[  5%] Generating standalone_crt/include/dmlc/json.h
[  5%] Generating standalone_crt/include/dmlc/logging.h
[  5%] Generating standalone_crt/include/dmlc/lua.h
[  6%] Generating standalone_crt/include/dmlc/memory.h
[  6%] Generating standalone_crt/include/dmlc/memory_io.h
[ 14%] Built target tvm_runtime_objs
[ 14%] Generating standalone_crt/include/dmlc/omp.h
[ 14%] Generating standalone_crt/include/dmlc/optional.h
[ 14%] Generating standalone_crt/include/dmlc/parameter.h
[ 15%] Generating standalone_crt/include/dmlc/registry.h
[ 15%] Generating standalone_crt/include/dmlc/recordio.h
[ 15%] Generating standalone_crt/include/dmlc/serializer.h
[ 15%] Generating standalone_crt/include/dmlc/strtonum.h
[ 15%] Generating standalone_crt/include/dmlc/thread_group.h
[ 15%] Generating standalone_crt/include/dmlc/thread_local.h
[ 16%] Generating standalone_crt/include/dmlc/threadediter.h
[ 16%] Generating standalone_crt/include/dmlc/timer.h
[ 16%] Generating standalone_crt/include/dmlc/type_traits.h
[ 16%] Generating standalone_crt/include/tvm/runtime/c_backend_api.h
[ 16%] Generating standalone_crt/include/tvm/runtime/crt/crt.h
[ 17%] Generating standalone_crt/include/tvm/runtime/crt/error_codes.h
[ 17%] Generating standalone_crt/include/tvm/runtime/c_runtime_api.h
[ 17%] Generating standalone_crt/include/tvm/runtime/crt/func_registry.h
[ 17%] Generating standalone_crt/include/tvm/runtime/crt/graph_executor_module.h
[ 17%] Generating standalone_crt/include/tvm/runtime/crt/graph_executor.h
[ 17%] Generating standalone_crt/include/tvm/runtime/crt/logging.h
[ 17%] Generating standalone_crt/include/tvm/runtime/crt/module.h
[ 18%] Generating standalone_crt/include/tvm/runtime/crt/packed_func.h
[ 18%] Generating standalone_crt/include/tvm/runtime/crt/memory.h
[ 18%] Generating standalone_crt/include/tvm/runtime/crt/platform.h
[ 18%] Generating standalone_crt/include/tvm/runtime/crt/rpc_common/framing.h
[ 18%] Generating standalone_crt/include/tvm/runtime/crt/rpc_common/frame_buffer.h
[ 19%] Generating standalone_crt/include/tvm/runtime/crt/rpc_common/session.h
[ 19%] Generating standalone_crt/include/tvm/runtime/crt/rpc_common/write_stream.h
[ 19%] Generating standalone_crt/include/tvm/runtime/crt/utvm_rpc_server.h
[ 19%] Generating standalone_crt/Makefile
[ 19%] Generating standalone_crt/include/tvm/runtime/crt/internal/common/func_registry.h
[ 19%] Generating standalone_crt/include/tvm/runtime/crt/internal/graph_executor/graph_executor.h
[ 20%] Generating standalone_crt/include/tvm/runtime/crt/internal/common/ndarray.h
[ 20%] Generating standalone_crt/include/tvm/runtime/crt/internal/memory/memory.h
[ 20%] Generating standalone_crt/include/tvm/runtime/crt/internal/graph_executor/load_json.h
[ 20%] Generating standalone_crt/src/runtime/crt/common/crt_backend_api.c
[ 20%] Generating standalone_crt/src/runtime/crt/common/crt_runtime_api.c
[ 21%] Generating standalone_crt/src/runtime/crt/common/func_registry.c
[ 21%] Generating standalone_crt/src/runtime/crt/common/ndarray.c
[ 21%] Generating standalone_crt/src/runtime/crt/common/packed_func.c
[ 21%] Generating standalone_crt/src/runtime/crt/graph_executor/graph_executor.c
[ 22%] Generating standalone_crt/src/runtime/crt/graph_executor_module/graph_executor_module.c
[ 22%] Generating standalone_crt/src/runtime/crt/graph_executor/load_json.c
[ 22%] Generating standalone_crt/template/host/crt_config.h
[ 22%] Generating standalone_crt/template/host/main.cc
[ 22%] Generating standalone_crt/src/runtime/crt/memory/memory.c
[ 22%] Generating standalone_crt/src/runtime/crt/utvm_rpc_common/frame_buffer.cc
[ 22%] Generating standalone_crt/src/runtime/crt/utvm_rpc_common/framing.cc
[ 22%] Generating standalone_crt/src/runtime/crt/utvm_rpc_common/write_stream.cc
[ 23%] Generating standalone_crt/src/runtime/crt/utvm_rpc_common/session.cc
[ 23%] Generating standalone_crt/src/runtime/crt/utvm_rpc_server/rpc_server.cc
[ 23%] Generating standalone_crt/src/runtime/minrpc/minrpc_server.h
[ 23%] Generating standalone_crt/src/runtime/minrpc/rpc_reference.h
[ 24%] Generating standalone_crt/src/support/generic_arena.h
[ 24%] Generating standalone_crt/template/crt_config-template.h
[ 24%] Built target standalone_crt
make[2]: Warning: File 'standalone_crt/template/crt_config-template.h' has modification time 0.76 s in the future
[ 24%] Generating host_standalone_crt/libmemory.a, host_standalone_crt/libgraph_executor.a, host_standalone_crt/libutvm_rpc_server.a, host_standalone_crt/libutvm_rpc_common.a, host_standalone_crt/libcommon.a
make[3]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
make[3]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
[ 24%] Building CXX object CMakeFiles/tvm_objs.dir/src/auto_scheduler/compute_dag.cc.o
[ 24%] Building CXX object CMakeFiles/tvm_objs.dir/src/auto_scheduler/search_policy/empty_policy.cc.o
[ 24%] Building CXX object CMakeFiles/tvm_objs.dir/src/auto_scheduler/search_policy/search_policy.cc.o

make[2]: warning:  Clock skew detected.  Your build may be incomplete.
[ 38%] Built target host_standalone_crt
[ 38%] Linking CXX shared library libtvm_runtime.so
In file included from /mnt/d/tvm/include/tvm/node/structural_equal.h:27:0,
                 from /mnt/d/tvm/include/tvm/node/reflection.h:26,
                 from /mnt/d/tvm/include/tvm/node/node.h:37,
                 from /mnt/d/tvm/include/tvm/auto_scheduler/transform_step.h:50,
                 from /mnt/d/tvm/include/tvm/auto_scheduler/loop_state.h:52,
                 from /mnt/d/tvm/src/auto_scheduler/search_policy/empty_policy.h:29,
                 from /mnt/d/tvm/src/auto_scheduler/search_policy/empty_policy.cc:26:
/mnt/d/tvm/include/tvm/runtime/container.h: In instantiation of ‘T tvm::runtime::Optional<T>::value() const [with T = tvm::runtime::Array<tvm::auto_scheduler::SearchCallback>]’:
/mnt/d/tvm/src/auto_scheduler/search_policy/empty_policy.cc:47:52:   required from here
/mnt/d/tvm/include/tvm/runtime/container.h:1625:3: internal compiler error: Segmentation fault
   }
   ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
CMakeFiles/tvm_objs.dir/build.make:354: recipe for target 'CMakeFiles/tvm_objs.dir/src/auto_scheduler/search_policy/empty_policy.cc.o' failed
make[2]: *** [CMakeFiles/tvm_objs.dir/src/auto_scheduler/search_policy/empty_policy.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 38%] Built target tvm_runtime
CMakeFiles/Makefile2:135: recipe for target 'CMakeFiles/tvm_objs.dir/all' failed
make[1]: *** [CMakeFiles/tvm_objs.dir/all] Error 2
Makefile:148: recipe for target 'all' failed
make: *** [all] Error 2
Is there an up-to-date uTVM code that I can refer to?
#2

I have switched the gcc version to gcc-8/g+±8 and now the segfault is gone. Writing this for future reference. I wonder if this was just me, or if something is wrong about the baseline VM, though.

#3

Yeah it is really interesting to see the GCC segfaults in container.h…

#4

Is constructing Optional<Array<SearchCallback>> causing the issue?

#5

Thanks for the help. Since the error is gone with g++8, I just moved on, so don’t know for sure where the issue stems anymore. It seemed like a too-deep rabbit hole for me to start looking into :slight_smile:

1 Like