One of the main challenges of running ML models on tiny devices is managing memory allocations at runtime. microTVM is also facing the same challenge and often it results in out of box tutorials/demos failures specially when they are targeted to run on new device. Ahead of time (AoT) compilation helped a lot with this issue by allocating most of the runtime memory requirements at compile time. In addition to AoT, Unified Static Memory Planing (USMP) work enabled a better approach to manage all memory allocations for model and runtime at compile time. At the current state of TVM if you build a microTVM project using AoT executor, if you build for a standalone project and enable USMP, it would require no memory allocation from user/developer since all the memory allocations are already handled at compile time. This is great but is not the default route when we build a microTVM project.
Since using USMP is critical to improve the microTVM developers experience, we propose to enable
USMP by default when
AoT Executor is selected and runtime is
C runtime (CRT). CRT is mostly used by microTVM use cases like CRT host, Zephyr, Arduino. This change would not affect other build flows in TVM.
I’m happy to discuss this change here and hear your feedbacks.