How to handle dynamic input shapes in TVM?

Questions
#1

Hi everyone,

I have a model that requires dynamic input shapes, but I’ve noticed that TVM optimizations work best with fixed input dimensions. I tried using relay.build with dynamic shapes, but performance seems to drop.

What’s the best approach to handle dynamic input shapes in TVM while maintaining efficiency?

Thanks!

#2

we are moving towards relax as the new graph IR, which handles dynamic shape dimensions. Note that when possible, having static shape is still desirable, but now relax can out of box handle dynamic dimensions in models like LLM Optimize Large Language Model — tvm 0.20.dev0 documentation contains some info about LLM specific items