[ONNX] from_onnx error when using SequenceEmpty in Loop

Troubleshooting
#1

Hi, I’m trying to run RNN-T in MLPerf Inference benchmark on TVM, so i convert the model from pytorch to onnx. However, i got some error when import the model, which seems same as the previous post.

It seems that Relay Loop operator can’t take an empty tuple as an input, which is generated by Onnx SequenceEmpty operator. and SequenceEmpty is generated by code section

l : List[torch.Tensor]=[]

in PyTorch.

I’ve looked for some workaround, but there are two points that I’m not sure.

  1. I’m not sure that representing ONNX Sequence as TVM tuple is good way. Is empty tuple supported in Relay? How to express the name of empty tuple? These questions are hard to answer for me
  2. Is the usage of List[torch.Tensor] a standard practice in DNNs? My understanding is that variable-length tensors might be represented via torch.cat or Relay concatenate.

Anyway, The error can be replicated by putting below code to tests/python/frontend/onnx/test_forward.py

@tvm.testing.parametrize_targets
def test_loop_sequence(target, dev):
"""test_empty_sequence"""

# Test creating an empty tensor sequence.
# generate loop that concat index
#   define loop body input
iter_count = helper.make_tensor_value_info("iter_count", TensorProto.INT64, [])
cond_in = helper.make_tensor_value_info("cond_in", TensorProto.BOOL, [])
seq_in = helper.make_tensor_sequence_value_info("seq_in", TensorProto.INT64, None)
index_in = helper.make_tensor_value_info("index_in", TensorProto.INT64, [])
#   define loop body output
cond_out = helper.make_tensor_value_info("cond_out", TensorProto.BOOL, [])
seq_out = helper.make_tensor_sequence_value_info("seq_out", TensorProto.INT64, None)
index_out = helper.make_tensor_value_info("index_out", TensorProto.INT64, [])
#   define loop nodes
const_true_inner_node = make_constant_node(
        "cond_out",
        TensorProto.BOOL,
        (),
        [True]
        ) 
const_1_node = make_constant_node(
        "const_one",
        onnx.TensorProto.INT64, (), [1]
        )
insert_node = helper.make_node(
    "SequenceInsert",
    inputs=["seq_in", "index_in"],
    outputs=["seq_out"],
)
add_node = helper.make_node(
    "Add",
    inputs=["index_in", "const_one"],
    outputs=["index_out"],
)
#   define loop_body_graph
loop_body = helper.make_graph(
    [
        const_true_inner_node,
        const_1_node,
        insert_node,
        add_node,
    ],
    "loop_body",
    [iter_count, cond_in, seq_in, index_in],
    [cond_out, seq_out, index_out],
)
# define main graph
#   define graph input
trip_count = helper.make_tensor_value_info("trip_count", TensorProto.INT64, [])
#   define graph output
res_seq = helper.make_tensor_sequence_value_info("res_seq", TensorProto.INT64, None)

const_true_node = make_constant_node(
        "cond",
        TensorProto.BOOL,
        (),
        [True]
        )
const_zero_node = make_constant_node(
        "index",
        TensorProto.INT64,
        (),
        [0]
        )
empty_node = helper.make_node(
    "SequenceEmpty",
    inputs=[],
    outputs=["empty_seq"],
    dtype=TensorProto.INT64
)
loop_node = helper.make_node(
    "Loop",
    inputs=["trip_count", "cond", "empty_seq", "index"],
    outputs=["res_seq", "res_index"],
    body=loop_body,
)

graph = helper.make_graph(
    [
        const_true_node,
        const_zero_node,
        empty_node,
        loop_node,
    ],
    "Sequence_loop_test",
    inputs=[trip_count],
    outputs=[res_seq],
)

model = helper.make_model(
    graph,
    producer_name="Sequence_loop_test",
)

verify_with_ort_with_inputs(model, [np.array(30).astype(np.int64)], target=target, dev=dev)