Unsupported ListType when importing RNN-T

Troubleshooting
#1

cc @gussmith23

I’m trying to import MLPerf’s RNN-T model into TVM through the Relay frontend. I’m facing similar issues to this thread, but the errors are different!

I was able to get a trace of the model, but after a call to relay.frontend.from_pytorch, I ran into a NotImplemented Error saying that The following operators are not implemented: ['aten::copy_'].

I was able to get aten::copy_ implemented by following the instructions in this thread, yielding an additional error: File "/root/tvm/python/tvm/relay/frontend/pytorch.py", line 2670, in lstm_layers raise NotImplementedError("Dropout for LSTM has not been supported yet!").

By uncommenting the two lines of implementation above line 2670, we were able to get past this error, finally reaching a new error on line 3768: NotImplementedError: Unsupported type: ListType.

I’d appreciate any advice for dealing with this new error!

#2

Can you try printing node before you hit NotImplementedError: Unsupported type: ListType?

The error is raised by a function that extracts constant from Torchscript IR node, and the error indicates that this model has a “constant list”, something we haven’t encountered in other models.

#3

Here’s what I got:

%324 : int[] = prim::Constant[value=[-1, 1, 1]]()

#4

If you can figure out a way to extract that list of values, we can probably do the same thing as https://github.com/apache/tvm/blob/a6a34046c432b3766e7c32bbd85c098812a12a68/python/tvm/relay/frontend/pytorch.py#L3752

#5

I can get the list of values of the node with values = node.output().toIValue().

I was able to get the model imported to tvm by appending a branch to the if statement.

A complete list of changes that I made can be seen in this commit of my tvm fork. If it would be useful, I’d love to help make a PR!

1 Like
#6

Amazing! I’ve been under an impression that this model is a poor fit for TVM. Does the model compile and output look ok?

I remember this model uses stft (short time fourier transform) in its preprocessing step, and manually rewritten python code that does decoding. So this model cannot be converted end to end.

Regarding the changes:

  • I wonder why dropout stuff in lstm is commented out right now. If you can demonstrate that your change is correct (by adding a test case), we should support this.
  • The workaround for aten::copy_: First, please make sure that the output is correct. We shouldn’t run the additional jit passes by default, since we cannot guarantee that we can always remove in-place mutation like this. But for users who know what they are doing, we can introduce an additional flag to run the passes as opt-in.