Encounter TVMError: "The source maps are not populated for this module."

Hi all,

I encountered the following error when doing relay.build. What’s the problem here? I did transform the original TF graph a little bit with converting a TVM-unsupported HashTableV2 node to a new HashTableV2 tvm node. But I didn’t change the shape_of op. The error happens in shape op:

def _shape():
    def _impl(inputs, attr, params, mod):
        is_symbolic_shape = False
        input_shape = _infer_shape(inputs[0], mod)  <<<<<<<<< failed in this line
        for axis in input_shape:
            if not isinstance(axis, (int, tvm.tir.IntImm)):
                is_symbolic_shape = True
                break

        if is_symbolic_shape:
            ret = _op.shape_of(inputs[0], dtype=attr["out_type"].name)
        else:
            ret = np.array(input_shape, dtype=attr["out_type"].name)
        return ret

    return _impl

The whole error dump:

  File "/data/tvm-0.7/python/tvm/relay/frontend/tensorflow2.py", line 956, in from_tensorflow
    gdef_lib=graph_def_library)
  File "/data/tvm-0.7/python/tvm/relay/frontend/tensorflow2.py", line 281, in from_tensorflow
    input_types=input_types)
  File "/data/tvm-0.7/python/tvm/relay/frontend/tensorflow2.py", line 449, in _get_relay_func
    self._backtrack_construct(graph, node.name)
  File "/data/tvm-0.7/python/tvm/relay/frontend/tensorflow2.py", line 655, in _backtrack_construct
    attr, node)
  File "/data/tvm-0.7/python/tvm/relay/frontend/tensorflow2.py", line 545, in _convert_operator
    self._module.mod)
  File "/data/tvm-0.7/python/tvm/relay/frontend/tensorflow_ops.py", line 2196, in _impl
    input_shape = _infer_shape(inputs[0], mod)
  File "/data/tvm-0.7/python/tvm/relay/frontend/common.py", line 515, in infer_shape
    out_type = infer_type(inputs, mod=mod)
  File "/data/tvm-0.7/python/tvm/relay/frontend/common.py", line 482, in infer_type
    mod = _transform.InferType()(mod)
  File "/data/tvm-0.7/python/tvm/ir/transform.py", line 161, in __call__
    return _ffi_transform_api.RunPass(self, mod)
  File "/data/tvm-0.7/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
    raise get_last_ffi_error()
tvm._ffi.base.TVMError: Traceback (most recent call last):
  9: TVMFuncCall
  8: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TypedPackedFunc<tvm::IRModule (tvm::transform::Pass, tvm::IRModule)>::AssignTypedLambda<tvm::transform::{lambda(tvm::transform::Pass, tvm::IRModule)#7}>(tvm::transform::{lambda(tvm::transform::Pass, tvm::IRModule)#7}, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
  7: tvm::transform::Pass::operator()(tvm::IRModule) const
  6: tvm::transform::Pass::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  5: tvm::transform::ModulePassNode::operator()(tvm::IRModule, tvm::transform::PassContext const&) const
  4: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TypedPackedFunc<tvm::IRModule (tvm::IRModule, tvm::transform::PassContext)>::AssignTypedLambda<tvm::relay::transform::InferType()::{lambda(tvm::IRModule, tvm::transform::PassContext const&)#1}>(tvm::relay::transform::InferType()::{lambda(tvm::IRModule, tvm::transform::PassContext const&)#1})::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
  3: tvm::DiagnosticContext::Render()
  2: tvm::DiagnosticRenderer::Render(tvm::DiagnosticContext const&)
  1: std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::TypedPackedFunc<void (tvm::DiagnosticContext)>::AssignTypedLambda<tvm::TerminalRenderer(std::ostream&)::{lambda(tvm::DiagnosticContext const&)#1}>(tvm::TerminalRenderer(std::ostream&)::{lambda(tvm::DiagnosticContext const&)#1})::{lambda(tvm::runtime::TVMArgs const&, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)
  0: tvm::ReportAt(tvm::DiagnosticContext const&, std::ostream&, tvm::Span const&, tvm::Diagnostic const&)
  File "/data/tvm-0.7/src/ir/diagnostic.cc", line 238
TVMError: The source maps are not populated for this module. Please use `tvm.relay.transform.AnnotateSpans` to attach source maps for error reporting. Error: The type inference pass was unable to infer a type for this expression.
This usually occurs when an operator call is under constrained in some way, check other reported errors for hints of what may of happened.

I added some printing in TVM diagnostic.cc It failed in some place not related with my change.

  // If the source name is not in the current source map, sources were not annotated.
  if (it == context->module->source_map->source_map.end()) {
    std::cout << "====DEBUG: ReportAt() source_name:" << span->source_name << std::endl;
    LOG(FATAL) << "The source maps are not populated for this module. "
               << "Please use `tvm.relay.transform.AnnotateSpans` to attach source maps for error "
                  "reporting. "
               << "Error: " << diagnostic->message;
  }

Here goes the result:

relay ir build...
====DEBUG: ReportAt() source_name:SourceName(StatefulPartitionedCall/functional_1/embed_plat/embedding_lookup/Identity, 0xce5dfb0)
Traceback (most recent call last):