Basically, I want to compile my DNN model (in PyTorch, ONNX, etc) with dynamic batch support. In other words, I want my compiled TVM module to process inputs with various batch sizes. For instance, I want my ResNet model to process inputs with sizes of [1, 3, 224, 224], [2, 3, 224, 224], and so on.
I’ve seen many similar topics, but no one clearly shows whether the dynamic batch support is possible or not. I want to know whether it is possible, and how can I exploit (i.e., compile with dynamic batch support)
I checked the code you attached and found that it might be possible if I build a DNN structure with Relay from the bottom. However, what I want is to compile the DNN graph which is already expressed as a graph in a framework (e.g., PyTorch, ONNX).
For example, how can I compile the TorchScript model with dynamic batch support like the code below?
Our PyTorch frontend does not support dynamic input shapes, since it is just too slow and not worth it for now. You can try ONNX frontend, see for example tvm/test_forward.py at 720e7b1ebd9b789a1100dee7536d0633c7941dd1 · apache/tvm · GitHub. You can use something like [relay.Any(), 3, 224, 224] as an input shape to pass to the shape dict.
I have one more question about dynamic input shape. The method you suggested is to exploit ‘vm’ executor instead of ‘graph’ executor. Are there any methods to support dynamic input shape with ‘graph’ executor?
No if you want to use dynamic shape, VM is always required. This is because the graph executor assumes that everything is static and preallocates all memory required based on static shape information.
Does autotvm or autoscheduler is available with dynamic batch configuration? I am using onnx and vm to support dynamic batch as you mention. But it seems like I cannot use autotvm or autoscheduler with dynamic batch input shape (e.g., input shape with relay.Any()). Are there any ways I can use them with dynamic batch support?
@masahi We check source code, find vm now support torch-script model and dynamic input shape. How about its performance compare with PyTorch(under python environment), is there any test examples for reference ? Thanks lot in advance.
Performance is expected to be extremely bad. We cannot tune any workload involving dynamic shapes, while PyTorch uses cuDNN etc that don’t have any issue with dynamic shapes.