Pass incompatibility: `AnnotateSpans` and `DefuseOps`

Troubleshooting
#1

I found that DefuseOps will not work if AnnotateSpans is applied at first. But if I recall correctly, AnnotateSpans only add some comments to Relay IR, so I guess such behaviour is not normal/intended?

To reproduce

import tvm
import tvm.relay as relay
from tvm.contrib import graph_executor
from tvm.relay import testing
import numpy as np

_RELAY_FUNCTION_HARD_PASSES_ = [ # Note these are types.
    # tvm.transform.PrintIR,
    relay.transform.AnnotateSpans,
    # tvm.transform.PrintIR,
    relay.transform.DefuseOps,
]

def example():
    out_channels = 32

    data = relay.var("data", relay.TensorType((1, 3, 128, 128), "float32"))
    weight = relay.var("weight")
    bn_gamma = relay.var("bn_gamma")
    bn_beta = relay.var("bn_beta")
    bn_mmean = relay.var("bn_mean")
    bn_mvar = relay.var("bn_var")

    simple_net = relay.nn.conv2d(
        data=data, weight=weight, kernel_size=(5, 5), channels=out_channels, padding=(1, 1)
    )
    simple_net = relay.nn.batch_norm(simple_net, bn_gamma, bn_beta, bn_mmean, bn_mvar)[0]
    simple_net = relay.Function(relay.analysis.free_vars(simple_net), simple_net)

    return testing.create_workload(simple_net)

if __name__ == '__main__':
    module, params = example()
    in_shape = module['main'].checked_type.arg_types[0].shape
    out_shape = module['main'].ret_type.shape

    # compile the model
    target = tvm.target.Target('cuda')
    dev = tvm.cuda()

    with tvm.transform.PassContext(opt_level=0):
        module, _ = relay.optimize(module, target, params)
    
    with tvm.transform.PassContext(opt_level=4):
        module = tvm.transform.Sequential(passes=[x() for x in _RELAY_FUNCTION_HARD_PASSES_], opt_level=4)(module)
    
    lib = relay.build(module, target=target, params=params)
    m = graph_executor.GraphModule(lib["default"](dev))

    data = np.zeros([int(x) for x in in_shape], dtype='float32')
    m.set_input('data', data)
    m.run()
    out = m.get_output(0, tvm.nd.empty(out_shape)).numpy()

    print(out.flatten()[0:10])

Error message

  File "/home/ljw/data/PROJECTS/tvm/python/tvm/ir/transform.py", line 161, in __call__
    return _ffi_transform_api.RunPass(self, mod)
  File "/home/ljw/data/PROJECTS/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
    raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
  71: TVMFuncCall
        at /data/ljw_data/PROJECTS/tvm/src/runtime/c_runtime_api.cc:474
  70: tvm::runtime::PackedFunc::CallPacked(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1151
  69: std::function<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
        at /home/ljw/bin/../lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
  68: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), void tvm::runtime::TypedPackedFunc<tvm::IRModule (tvm::transform::Pass, tvm::IRModule)>::AssignTypedLambda<tvm::transform::$_6>(tvm::transform::$_6, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
        at /home/ljw/bin/../lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
  67: void tvm::runtime::TypedPackedFunc<tvm::IRModule (tvm::transform::Pass, tvm::IRModule)>::AssignTypedLambda<tvm::transform::$_6>(tvm::transform::$_6, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1480
  66: void tvm::runtime::detail::unpack_call<tvm::IRModule, 2, tvm::transform::$_6>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, tvm::transform::$_6 const&, tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1421
  65: void tvm::runtime::detail::unpack_call_dispatcher<tvm::IRModule, 2, 0, tvm::transform::$_6>::run<>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, tvm::transform::$_6 const&, tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1382
  64: void tvm::runtime::detail::unpack_call_dispatcher<tvm::IRModule, 1, 1, tvm::transform::$_6>::run<tvm::runtime::TVMMovableArgValueWithContext_>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, tvm::transform::$_6 const&, tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*, tvm::runtime::TVMMovableArgValueWithContext_&&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1382
  63: void tvm::runtime::detail::unpack_call_dispatcher<tvm::IRModule, 0, 2, tvm::transform::$_6>::run<tvm::runtime::TVMMovableArgValueWithContext_, tvm::runtime::TVMMovableArgValueWithContext_>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, tvm::transform::$_6 const&, tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*, tvm::runtime::TVMMovableArgValueWithContext_&&, tvm::runtime::TVMMovableArgValueWithContext_&&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1397
  62: tvm::transform::$_6::operator()(tvm::transform::Pass, tvm::IRModule) const
        at /data/ljw_data/PROJECTS/tvm/src/ir/transform.cc:525
  61: tvm::transform::Pass::operator()(tvm::IRModule) const
        at /data/ljw_data/PROJECTS/tvm/src/ir/transform.cc:255
  60: tvm::transform::Pass::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
        at /data/ljw_data/PROJECTS/tvm/src/ir/transform.cc:267
  59: tvm::transform::SequentialNode::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
        at /data/ljw_data/PROJECTS/tvm/src/ir/transform.cc:481
  58: tvm::transform::Pass::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
        at /data/ljw_data/PROJECTS/tvm/src/ir/transform.cc:267
  57: tvm::relay::transform::FunctionPassNode::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
        at /data/ljw_data/PROJECTS/tvm/src/relay/ir/transform.cc:145
  56: tvm::runtime::TypedPackedFunc<tvm::relay::Function (tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext)>::operator()(tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1498
  55: tvm::relay::Function tvm::runtime::detail::typed_packed_call_dispatcher<tvm::relay::Function>::run<tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext>(tvm::runtime::PackedFunc const&, tvm::relay::Function&&, tvm::IRModule&&, tvm::transform::PassContext&&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1444
  54: tvm::runtime::TVMRetValue tvm::runtime::PackedFunc::operator()<tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext>(tvm::relay::Function&&, tvm::IRModule&&, tvm::transform::PassContext&&) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1369
  53: std::function<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)>::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const
        at /home/ljw/bin/../lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706
  52: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), void tvm::runtime::TypedPackedFunc<tvm::relay::Function (tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext)>::AssignTypedLambda<tvm::relay::transform::DefuseOps()::$_0>(tvm::relay::transform::DefuseOps()::$_0)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
        at /home/ljw/bin/../lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316
  51: void tvm::runtime::TypedPackedFunc<tvm::relay::Function (tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext)>::AssignTypedLambda<tvm::relay::transform::DefuseOps()::$_0>(tvm::relay::transform::DefuseOps()::$_0)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}::operator()(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1492
  50: void tvm::runtime::detail::unpack_call<tvm::relay::Function, 3, tvm::relay::transform::DefuseOps()::$_0>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, tvm::relay::transform::DefuseOps()::$_0 const&, tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1421
  49: void tvm::runtime::detail::unpack_call_dispatcher<tvm::relay::Function, 3, 0, tvm::relay::transform::DefuseOps()::$_0>::run<>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, tvm::relay::transform::DefuseOps()::$_0 const&, tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1382
  48: void tvm::runtime::detail::unpack_call_dispatcher<tvm::relay::Function, 2, 1, tvm::relay::transform::DefuseOps()::$_0>::run<tvm::runtime::TVMMovableArgValueWithContext_>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, tvm::relay::transform::DefuseOps()::$_0 const&, tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*, tvm::runtime::TVMMovableArgValueWithContext_&&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1382
  47: void tvm::runtime::detail::unpack_call_dispatcher<tvm::relay::Function, 1, 2, tvm::relay::transform::DefuseOps()::$_0>::run<tvm::runtime::TVMMovableArgValueWithContext_, tvm::runtime::TVMMovableArgValueWithContext_>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, tvm::relay::transform::DefuseOps()::$_0 const&, tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*, tvm::runtime::TVMMovableArgValueWithContext_&&, tvm::runtime::TVMMovableArgValueWithContext_&&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1382
  46: void tvm::runtime::detail::unpack_call_dispatcher<tvm::relay::Function, 0, 3, tvm::relay::transform::DefuseOps()::$_0>::run<tvm::runtime::TVMMovableArgValueWithContext_, tvm::runtime::TVMMovableArgValueWithContext_, tvm::runtime::TVMMovableArgValueWithContext_>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, tvm::relay::transform::DefuseOps()::$_0 const&, tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*, tvm::runtime::TVMMovableArgValueWithContext_&&, tvm::runtime::TVMMovableArgValueWithContext_&&, tvm::runtime::TVMMovableArgValueWithContext_&&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/runtime/packed_func.h:1397
  45: tvm::relay::transform::DefuseOps()::$_0::operator()(tvm::relay::Function, tvm::IRModule, tvm::transform::PassContext) const
        at /data/ljw_data/PROJECTS/tvm/src/relay/transforms/defuse_ops.cc:83
  44: tvm::relay::DefuseOps(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/src/relay/transforms/defuse_ops.cc:77
  43: tvm::relay::ExprMutator::Mutate(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:189
  42: tvm::relay::ExprMutator::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/src/relay/ir/expr_functor.cc:154
  41: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:94
  40: tvm::NodeFunctor<tvm::RelayExpr (tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/node/functor.h:97
  39: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#5}::__invoke(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:126
  38: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#5}::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:126
  37: tvm::relay::ExprMutator::VisitExpr_(tvm::relay::FunctionNode const*)
        at /data/ljw_data/PROJECTS/tvm/src/relay/ir/expr_functor.cc:212
  36: tvm::relay::ExprMutator::Mutate(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:189
  35: tvm::relay::ExprMutator::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/src/relay/ir/expr_functor.cc:154
  34: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:94
  33: tvm::NodeFunctor<tvm::RelayExpr (tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/node/functor.h:97
  32: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#6}::__invoke(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:127
  31: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#6}::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:127
  30: tvm::relay::DefuseOpsMutator::VisitExpr_(tvm::relay::CallNode const*)
        at /data/ljw_data/PROJECTS/tvm/src/relay/transforms/defuse_ops.cc:64
  29: tvm::relay::ExprMutator::VisitExpr_(tvm::relay::CallNode const*)
        at /data/ljw_data/PROJECTS/tvm/src/relay/ir/expr_functor.cc:235
  28: tvm::relay::ExprMutator::Mutate(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:189
  27: tvm::relay::ExprMutator::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/src/relay/ir/expr_functor.cc:154
  26: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:94
  25: tvm::NodeFunctor<tvm::RelayExpr (tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/node/functor.h:97
  24: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#6}::__invoke(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:127
  23: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#6}::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:127
  22: tvm::relay::DefuseOpsMutator::VisitExpr_(tvm::relay::CallNode const*)
        at /data/ljw_data/PROJECTS/tvm/src/relay/transforms/defuse_ops.cc:64
  21: tvm::relay::ExprMutator::VisitExpr_(tvm::relay::CallNode const*)
        at /data/ljw_data/PROJECTS/tvm/src/relay/ir/expr_functor.cc:235
  20: tvm::relay::ExprMutator::Mutate(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:189
  19: tvm::relay::ExprMutator::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/src/relay/ir/expr_functor.cc:154
  18: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:94
  17: tvm::NodeFunctor<tvm::RelayExpr (tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/node/functor.h:97
  16: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#6}::__invoke(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:127
  15: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#6}::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:127
  14: tvm::relay::DefuseOpsMutator::VisitExpr_(tvm::relay::CallNode const*)
        at /data/ljw_data/PROJECTS/tvm/src/relay/transforms/defuse_ops.cc:69
  13: tvm::relay::ExprMutator::Mutate(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:189
  12: tvm::relay::ExprMutator::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/src/relay/ir/expr_functor.cc:154
  11: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:94
  10: tvm::NodeFunctor<tvm::RelayExpr (tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/node/functor.h:97
  9: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#6}::__invoke(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:127
  8: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#6}::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:127
  7: tvm::relay::ExprMutator::VisitExpr_(tvm::relay::CallNode const*)
        at /data/ljw_data/PROJECTS/tvm/src/relay/ir/expr_functor.cc:235
  6: tvm::relay::ExprMutator::Mutate(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:189
  5: tvm::relay::ExprMutator::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/src/relay/ir/expr_functor.cc:154
  4: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::VisitExpr(tvm::RelayExpr const&)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:94
  3: tvm::NodeFunctor<tvm::RelayExpr (tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)>::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/node/functor.h:97
  2: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#3}::__invoke(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:124
  1: tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>::InitVTable()::{lambda(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*)#3}::operator()(tvm::runtime::ObjectRef const&, tvm::relay::ExprFunctor<tvm::RelayExpr (tvm::RelayExpr const&)>*) const
        at /data/ljw_data/PROJECTS/tvm/include/tvm/relay/expr_functor.h:124
  0: tvm::relay::DefuseOpsMutator::FuncBodyMutator::VisitExpr_(tvm::relay::VarNode const*)
        at /data/ljw_data/PROJECTS/tvm/src/relay/transforms/defuse_ops.cc:54
  File "/data/ljw_data/PROJECTS/tvm/src/relay/transforms/defuse_ops.cc", line 54
TVMError: 
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (id >= 0 && size_t(id) < args_.size()) is false:

Debugging: IR content before/after AnnotateSpans

See: Saved diff eLMrnxCU - Diff Checker

The only difference is “GeneratedSource” inserted to the comment.

image
image2380Ă—254 69.6 KB

Debugging: data content when crashed

    Expr VisitExpr_(const VarNode* n) {
      const std::string& name = n->name_hint(); 
      // `name` here is "p02"
      ICHECK(!name.empty() && (name[0] == 'p'));
      std::string id_str = name.substr(1);
      // `id_str` is 02, and `id` is `2`
      int id = std::stoi(id_str);
      ICHECK(id >= 0 && size_t(id) < args_.size()); // crashed here! b.c. size_t(id) = 2 < args_.size() = 2
      // `args_` is "[Var(data, ty=TensorType([1, 3, 128, 128], float32)), Constant([[[[ ..." (conv weight omitted)
      return args_[id];
    }
1 Like
#2

@comaniac is familiar with these two passes :slight_smile:

1 Like
#3

Interesting. If commenting out the following two lines, the error goes away somehow… I thought opt_level=0 pass only does type inference (via InferType), not sure why applying it aforehand causes the crash.

with tvm.transform.PassContext(opt_level=0):
        module, _ = relay.optimize(module, target, params)
#4

Ah what I am familiar with is AnnotateTarget, I never looked into AnnotateSpans tho.

opt=0 should also do some optimizations IIUC, so it’s hard to judge only by that.

1 Like
#5

Yeah. I turned on the debug log, and find several level-1 passes (including FuseOps, InlineGlobals, LabelOps) were executed, even though opt_level=0 is specified in PassContext.

1 Like
#6

Hi, @comaniac @yuchenj @junrushao thanks you guys for the quick reply!

If commenting out the following two lines, the error goes away somehow…

Sure, because if we don’t apply some O0 optimization passes, the IRModule looks so simple that passes like DefuseOps will not affect that. However, if you uncomment relay.transform.AnnotateSpans, the error goes away as well, which means it triggers a bug by putting relay.transform.AnnotateSpans in front of DefuseOps… And as I mentioned, the relay.transform.AnnotateSpans pass only put some comment messages (I am not sure if some optimization passes will also look at comment information) so I doubt if this is a potential bug… Will check deeper about this and share with you any updates.

1 Like
#7

Well, I found that the IRs before and after transformation only has differences on comments, but the metadata are quite different. Maybe it has something to do with the content in metadata…

2 Likes
#8

I finally figured out why. @junrushao @yuchenj

AnnotateSpans is to add span information to an IR. However, its implementation is very simple:

  1. print current IR using AsText and
  2. parse it again to obtain span information.

Since when printing an IRModule it is ill-formed to have variables in the same names, when calling AsText, GetUniqueName will be applied to modified in-memory names to some other names making sure that the whole printed IR do not have identical variable names. For example, if that current IR has 2 functions whose parameters are both “p0”. AsText will rename one of them like “p01”.

Hence, step 1 of AnnotateSpans might change the name of variables. And Step 2 (parsing) directly read those variables with their new names, breaking original information about their old names which might contain some kind of information itself (e.g., FuseOps). And this is the root cause making AnnotateSpans incompatible with passes like DefuseOps which looks at the variable name:

    Expr VisitExpr_(const VarNode* n) {
      const std::string& name = n->name_hint();
      ICHECK(!name.empty() && (name[0] == 'p')); // <--- `AnnotateSpans` breaks the name so that things become illegal!
      std::string id_str = name.substr(1);
      int id = std::stoi(id_str);
      ICHECK(id >= 0 && size_t(id) < args_.size());
      return args_[id];
    }

A quick fix is to record a reverse mapping when GetUniqueName changes the names (new_name → old_name). And after applying step 1&2, we resume the names of the new IR module through renaming.

Also thanks for the help from @merrymercy

2 Likes
#9

That’s super cool to find the root cause!

1 Like