[Potential Bug] TVMError: Check failed: !b.is_const(0): divide by zero

sqchao · December 16, 2021, 1:57pm

Describtion:

The compilation process crashed when I optimizad&compiled a specific model using vm with opt_level = 2.

Trigger conditions:

opt_level = 2 or 3
using ex = relay.create_executor("vm", mod=mod, target='llvm') or vm_exec = relay.vm.compile(mod, target='llvm') to compile the model.

Related script

...
with tvm.transform.PassContext(opt_level=2):
    #intrp = relay.build_module.create_executor('graph', mod, tvm.cpu(0), 'llvm')
    ex = relay.create_executor("vm", mod=mod, target='llvm')

Notices:

opt_level = 1 can’t trigger this crash.
intrp = relay.build_module.create_executor('graph', mod, tvm.cpu(0), 'llvm') statement can’t trigger this crash.

Crash Message:

Traceback (most recent call last):
  File "tvm_graph_7.py", line 54, in <module>
    ex = relay.create_executor("vm", mod=mod, target='llvm')
  File "/workplace/software/tvm/tvm8/python/tvm/relay/build_module.py", line 424, in create_executor
    return VMExecutor(mod, ctx, target)
  File "/workplace/software/tvm/tvm8/python/tvm/relay/backend/vm.py", line 257, in __init__
    self.executable = compile(mod, target)
  File "/workplace/software/tvm/tvm8/python/tvm/relay/backend/vm.py", line 68, in compile
    compiler.lower(mod, target, target_host)
  File "/workplace/software/tvm/tvm8/python/tvm/relay/backend/vm.py", line 135, in lower
    self._lower(mod, target, target_host)
  File "tvm/_ffi/_cython/./packed_func.pxi", line 321, in tvm._ffi._cy3.core.PackedFuncBase.__call__
  File "tvm/_ffi/_cython/./packed_func.pxi", line 256, in tvm._ffi._cy3.core.FuncCall
  File "tvm/_ffi/_cython/./packed_func.pxi", line 245, in tvm._ffi._cy3.core.FuncCall3
  File "tvm/_ffi/_cython/./base.pxi", line 160, in tvm._ffi._cy3.core.CALL
tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (8) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::NodeFunctor<void (tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<void (tvm::RelayExpr const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<void (tvm::RelayExpr const&)>*) const+0x3a2) [0x7f83989e9bd2]
  [bt] (7) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::relay::vm::VMFunctionCompiler::VisitExpr_(tvm::relay::CallNode const*)+0x1683) [0x7f8398fca8a3]
  [bt] (6) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::relay::OpMatch<void>::operator()(tvm::relay::Call const&)+0x376) [0x7f8398fd03e6]
  [bt] (5) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::relay::vm::VMFunctionCompiler::VisitExpr_(tvm::relay::CallNode const*)::{lambda(tvm::runtime::Array<tvm::RelayExpr, void> const&, tvm::Attrs const&, tvm::runtime::Array<tvm::Type, void> const&)#1}::operator()(tvm::runtime::Array<tvm::RelayExpr, void> const&, tvm::Attrs const&, tvm::runtime::Array<tvm::Type, void> const&) const+0x3b1) [0x7f8398fd2501]
  [bt] (4) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::relay::vm::VMFunctionCompiler::EmitInvokeTVMOp(tvm::relay::Function const&, tvm::RelayExpr const&, tvm::RelayExpr const&)+0x231e) [0x7f8398fd4a1e]
  [bt] (3) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::relay::CompileEngineImpl::Lower(tvm::relay::CCacheKey const&)+0x1a) [0x7f8398ec211a]
  [bt] (2) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::relay::CompileEngineImpl::LowerInternal(tvm::relay::CCacheKey const&)+0x20aa) [0x7f8398ec824a]
  [bt] (1) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::runtime::TVMRetValue tvm::runtime::PackedFunc::operator()<tvm::te::Schedule const&, tvm::runtime::Array<tvm::te::Tensor, void>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, tvm::relay::Function const&>(tvm::te::Schedule const&, tvm::runtime::Array<tvm::te::Tensor, void>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, tvm::relay::Function const&) const+0x18e) [0x7f8398eca73e]
  [bt] (0) /workplace/software/tvm/tvm8/build/libtvm.so(+0x4b42e6b) [0x7f8399372e6b]
  File "/workplace/software/tvm/tvm8/python/tvm/relay/backend/_backend.py", line 49, in lower
    f = tvm.driver.lower(sch, inputs, name=func_name)
  File "/workplace/software/tvm/tvm8/python/tvm/driver/build_module.py", line 207, in lower
    mod = optimize(mod)
  File "/workplace/software/tvm/tvm8/python/tvm/ir/transform.py", line 127, in __call__
    return _ffi_transform_api.RunPass(self, mod)
  File "tvm/_ffi/_cython/./packed_func.pxi", line 321, in tvm._ffi._cy3.core.PackedFuncBase.__call__
  File "tvm/_ffi/_cython/./packed_func.pxi", line 256, in tvm._ffi._cy3.core.FuncCall
  File "tvm/_ffi/_cython/./packed_func.pxi", line 245, in tvm._ffi._cy3.core.FuncCall3
  File "tvm/_ffi/_cython/./base.pxi", line 160, in tvm._ffi._cy3.core.CALL
  [bt] (8) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::tir::DataTypeRewriter::operator()(tvm::tir::Stmt)+0x49) [0x7f8397f02ca9]
  [bt] (7) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::NodeFunctor<void (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<void (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<void (tvm::tir::Stmt const&)>*) const+0x3a2) [0x7f8396f73882]
  [bt] (6) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::tir::StmtVisitor::VisitStmt_(tvm::tir::StoreNode const*)+0x2f) [0x7f8397c896ef]
  [bt] (5) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::tir::DataTypeVisitor::VisitExpr(tvm::PrimExpr const&)+0x19d) [0x7f8397f0963d]
  [bt] (4) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::arith::ConstIntBoundAnalyzer::operator()(tvm::PrimExpr const&, std::unordered_map<tvm::PrimExpr, tvm::arith::ConstIntBound, tvm::runtime::ObjectPtrHash, tvm::runtime::ObjectPtrEqual, std::allocator<std::pair<tvm::PrimExpr const, tvm::arith::ConstIntBound> > >*)+0x31) [0x7f8396f214c1]
  [bt] (3) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::arith::ConstIntBoundAnalyzer::Impl::VisitExpr(tvm::PrimExpr const&)+0x24) [0x7f8396f28184]
  [bt] (2) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::NodeFunctor<tvm::arith::ConstIntBoundAnalyzer::Entry (tvm::runtime::ObjectRef const&, tvm::tir::ExprFunctor<tvm::arith::ConstIntBoundAnalyzer::Entry (tvm::PrimExpr const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::ExprFunctor<tvm::arith::ConstIntBoundAnalyzer::Entry (tvm::PrimExpr const&)>*) const+0x3a2) [0x7f8396f2a732]
  [bt] (1) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::arith::ConstIntBoundAnalyzer::Impl::VisitExpr_(tvm::tir::DivNode const*)+0x21a) [0x7f8396f3e07a]
  [bt] (0) /workplace/software/tvm/tvm8/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x11c) [0x7f8396e9acdc]
  File "/workplace/software/tvm/tvm8/src/arith/const_int_bound.cc", line 214
  File "tvm/_ffi/_cython/./packed_func.pxi", line 55, in tvm._ffi._cy3.core.tvm_callback
  File "/workplace/software/tvm/tvm8/python/tvm/relay/backend/_backend.py", line 57, in lower
    raise RuntimeError(msg)
  File "/workplace/software/tvm/tvm8/python/tvm/relay/backend/_backend.py", line 49, in lower
    f = tvm.driver.lower(sch, inputs, name=func_name)
  File "/workplace/software/tvm/tvm8/python/tvm/driver/build_module.py", line 207, in lower
    mod = optimize(mod)
  File "/workplace/software/tvm/tvm8/python/tvm/ir/transform.py", line 127, in __call__
    return _ffi_transform_api.RunPass(self, mod)
  File "tvm/_ffi/_cython/./packed_func.pxi", line 321, in tvm._ffi._cy3.core.PackedFuncBase.__call__
  File "tvm/_ffi/_cython/./packed_func.pxi", line 256, in tvm._ffi._cy3.core.FuncCall
  File "tvm/_ffi/_cython/./packed_func.pxi", line 245, in tvm._ffi._cy3.core.FuncCall3
  File "tvm/_ffi/_cython/./base.pxi", line 160, in tvm._ffi._cy3.core.CALL
  [bt] (8) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::tir::DataTypeRewriter::operator()(tvm::tir::Stmt)+0x49) [0x7f8397f02ca9]
  [bt] (7) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::NodeFunctor<void (tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<void (tvm::tir::Stmt const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::StmtFunctor<void (tvm::tir::Stmt const&)>*) const+0x3a2) [0x7f8396f73882]
  [bt] (6) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::tir::StmtVisitor::VisitStmt_(tvm::tir::StoreNode const*)+0x2f) [0x7f8397c896ef]
  [bt] (5) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::tir::DataTypeVisitor::VisitExpr(tvm::PrimExpr const&)+0x19d) [0x7f8397f0963d]
  [bt] (4) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::arith::ConstIntBoundAnalyzer::operator()(tvm::PrimExpr const&, std::unordered_map<tvm::PrimExpr, tvm::arith::ConstIntBound, tvm::runtime::ObjectPtrHash, tvm::runtime::ObjectPtrEqual, std::allocator<std::pair<tvm::PrimExpr const, tvm::arith::ConstIntBound> > >*)+0x31) [0x7f8396f214c1]
  [bt] (3) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::arith::ConstIntBoundAnalyzer::Impl::VisitExpr(tvm::PrimExpr const&)+0x24) [0x7f8396f28184]
  [bt] (2) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::NodeFunctor<tvm::arith::ConstIntBoundAnalyzer::Entry (tvm::runtime::ObjectRef const&, tvm::tir::ExprFunctor<tvm::arith::ConstIntBoundAnalyzer::Entry (tvm::PrimExpr const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::tir::ExprFunctor<tvm::arith::ConstIntBoundAnalyzer::Entry (tvm::PrimExpr const&)>*) const+0x3a2) [0x7f8396f2a732]
  [bt] (1) /workplace/software/tvm/tvm8/build/libtvm.so(tvm::arith::ConstIntBoundAnalyzer::Impl::VisitExpr_(tvm::tir::DivNode const*)+0x21a) [0x7f8396f3e07a]
  [bt] (0) /workplace/software/tvm/tvm8/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x11c) [0x7f8396e9acdc]
  File "/workplace/software/tvm/tvm8/src/arith/const_int_bound.cc", line 214
TVMError: Check failed: !b.is_const(0): divide by zero
During handling of the above exception, another exception occurred:

TVMError: Check failed: !b.is_const(0): divide by zero
Error during compile function
-----------------------------
#[version = "0.0.5"]
fn (%p0: int64, Primitive=1) -> int64 {
  divide(%p0, 0 /* ty=int64 */) /* ty=int64 */
}

Questions:

Is it a bug inner TVM?
Why can compilation by vm trigger this crash, but relay.build_module.create_executor can’t? What’s the difference between the 2 compilation methods?

Wish your comments, Thanks in advance.

sqchao · December 17, 2021, 3:06pm

@merrymercy @tqchen @FrozenGene Could your give me some advice about this bug? Thanks in advance.