Error with relay.Any()

Troubleshooting
#1

Dear All

I can use VM runtime to do inference with batch size = 1

but I would like to do the dynamic batch size

so I set the batch dimension to tvm.relay.Any()

But got the following error

Output : src/tvm/python/tvm/relay/frontend/pytorch.py", line 4237, in get_relay_ty raise RuntimeError(msg) RuntimeError: Shape for Tensors must be lists of ints

input_shape[0] = tvm.relay.Any()
input_name = "input"
shape_list = [(input_name, (input_shape, dtype))]
mod, params = relay.frontend.from_pytorch(s, shape_list, default_dtype = dtype)
#2

Our PT frontend doesn’t support dynamic input shape. It can be supported, but the performance would be bad anyway, since we cannot tune on dynamic workload.

#3

Dear Masahi

Thanks for your reply

Will Pytorch Front End support dynamic batch inference in the future ?

I am doing it on GPU and would consider performance (FPS) the most important point ( though I am thinking to implement on different hardware in the future, currently I am focusing on GPU )

What would you recommend for the solution, or which solution would likely have the highest FPS?

Solution 1

Convert Pytorch model to ONNX model, compile in TVM and Run in TVM Virtual Machine for Batch inference

( Could I tune the Dynamic Batch Size model with ONNX model in TVM with Virtual Machine inference? )

Solution 2

Use PyTorch (Libtorch ) directly with Batch inference

Solution3

Use TVM + VM + TensorRT + Pytorch Model

Since the dynamic size is not tunable

Is it ok to use TensorRT in Virtual Machine Runtime with Dynamic Batch Inference?

#4

It would be easy to support dynamic input shape to the PT frontend, we haven’t done that because it’s not worth it (dynamic shape inference being too slow in TVM currently). If you need dynamic batch support, you are better off using PyTorch with or without TRT (for now).

That said, there is an ongoing work on adding dynamic shape tuning support to TVM. I don’t know when it will become ready.