Dynamic batch (input) support

agongee · May 24, 2021, 9:59am

Hello.

Basically, I want to compile my DNN model (in PyTorch, ONNX, etc) with dynamic batch support. In other words, I want my compiled TVM module to process inputs with various batch sizes. For instance, I want my ResNet model to process inputs with sizes of [1, 3, 224, 224], [2, 3, 224, 224], and so on.

I’ve seen many similar topics, but no one clearly shows whether the dynamic batch support is possible or not. I want to know whether it is possible, and how can I exploit (i.e., compile with dynamic batch support)

Thanks a lot.

masahi · May 24, 2021, 11:03am

It’s possible, but I don’t recommend it since we cannot autotune dynamic batch workloads and hence things would be extremely slow.

See for example,

github.com

apache/tvm/blob/720e7b1ebd9b789a1100dee7536d0633c7941dd1/tests/python/relay/test_any.py#L518




    targets = None
    if use_cudnn and tvm.get_global_func("tvm.contrib.cudnn.conv.output_shape", True):
        targets = [("cuda -libs=cudnn", tvm.cuda(0))]


    check_result([data_np, kernel_np], mod, ref_out_shape, assert_shape=True, targets=targets)




# TODO(@kevinthesun): Support dynamic input height and width.
@tvm.testing.uses_gpu
def test_any_conv2d():
    verify_any_conv2d(
        (relay.Any(), 64, 224, 224),
        (64, 64, 3, 3),
        (1, 1),
        (1, 1),
        (1, 1),
        (1, 64, 224, 224),
        (1, 64, 224, 224),
    )
    verify_any_conv2d(

agongee · May 24, 2021, 11:20am

Thanks for your kind reply.

I checked the code you attached and found that it might be possible if I build a DNN structure with Relay from the bottom. However, what I want is to compile the DNN graph which is already expressed as a graph in a framework (e.g., PyTorch, ONNX).

For example, how can I compile the TorchScript model with dynamic batch support like the code below?

github.com

apache/tvm/blob/main/tutorials/frontend/from_pytorch.py#L97


)
img = my_preprocess(img)
img = np.expand_dims(img, 0)


######################################################################
# Import the graph to Relay
# -------------------------
# Convert PyTorch graph to Relay graph. The input name can be arbitrary.
input_name = "input0"
shape_list = [(input_name, img.shape)]
mod, params = relay.frontend.from_pytorch(scripted_model, shape_list)


######################################################################
# Relay Build
# -----------
# Compile the graph to llvm target with given input specification.
target = tvm.target.Target("llvm", host="llvm")
dev = tvm.cpu(0)
with tvm.transform.PassContext(opt_level=3):
    lib = relay.build(mod, target=target, params=params)

Thanks a lot.

masahi · May 24, 2021, 11:57am

Our PyTorch frontend does not support dynamic input shapes, since it is just too slow and not worth it for now. You can try ONNX frontend, see for example tvm/test_forward.py at 720e7b1ebd9b789a1100dee7536d0633c7941dd1 · apache/tvm · GitHub. You can use something like [relay.Any(), 3, 224, 224] as an input shape to pass to the shape dict.

agongee · May 26, 2021, 3:24am

Thanks for your great help.

I have one more question about dynamic input shape. The method you suggested is to exploit ‘vm’ executor instead of ‘graph’ executor. Are there any methods to support dynamic input shape with ‘graph’ executor?

masahi · May 26, 2021, 5:09am

No if you want to use dynamic shape, VM is always required. This is because the graph executor assumes that everything is static and preallocates all memory required based on static shape information.

agongee · May 29, 2021, 3:37pm

Hi,

Does autotvm or autoscheduler is available with dynamic batch configuration? I am using onnx and vm to support dynamic batch as you mention. But it seems like I cannot use autotvm or autoscheduler with dynamic batch input shape (e.g., input shape with relay.Any()). Are there any ways I can use them with dynamic batch support?

Thanks a lot!

masahi · May 29, 2021, 7:27pm

No, that’s what I said above and why I don’t recommend using dynamic batch for now.

delldu · June 26, 2021, 4:20pm

@masahi We check source code, find vm now support torch-script model and dynamic input shape. How about its performance compare with PyTorch(under python environment), is there any test examples for reference ? Thanks lot in advance.

masahi · June 26, 2021, 11:58pm

Performance is expected to be extremely bad. We cannot tune any workload involving dynamic shapes, while PyTorch uses cuDNN etc that don’t have any issue with dynamic shapes.

jcf94 · June 28, 2021, 5:41am

Dynamic shape support has been an important topic for TVM for a long time. Currently VM is the only way to process a dynamic model.

For the AutoScheduler, we’ve had many discussions about it while there’s no perfect approach to solve such problem now.

p.s. @comaniac may have some experience on partial dynamic shape support for auto tuning, but I believe the code is not ready now.