I am currently working on a project where I need to handle dynamic shapes (in my case dynamic batch sizes) with a ONNX model. I saw in mid-2021 that Auto Scheduler didn’t handle Relay.Any() and future work needed to be done. The workaround I chose is optimizing the model after fixing the batch size, and this for multiple batch sizes, and then loading the corresponding exported library (.so file) like this everytime I need to switch batch size. However, it takes a lot of time (as tuning one model can take multiple hours). Furthermore, I have memory (RAM and disk) limitations so the workaround is a bit problematic.
Are there another way to handle dynamic shapes or do dynamic shapes be supported soon?
Thank you!
Yes. @yuchenj is working on the next-generational high-level IR called Relax that is able to represent dynamic shapes properly, and Bojian Zheng (UofT) and @ziheng are working on dynamic shape tuning called DietCode
Hi @Upscale4152, currently we are doing open development for Relax in a separate repo: Relax repo. We will release a Relax Roadmap RFC this week as a start to discuss the upstreaming plan with the broader TVM community.
This Jupyter notebook contains a set of demos showing how Relax handles dynamic-shape workloads, feel free to try it out!
For more details about Relax, please check out the Relax pre-RFC, the TVM Community Meeting from 04/20/2022 here, the TVMCon 2021 talk here, and design docs here.