Several questions about micro TVM

[areusch]

hi @sho apologies for the delay as we were out on vacation for an extended weekend.

some answers to your questions:

@sho

Where is ‘micro tvm’ in this image? I’m not sure what ‘micro tvm’ exactly means, and how it’s involved in the process of generating binary for MCUs. If ‘micro tvm’ means the mechanism or something to generate binary and doesn’t mean the code or binary that will be stored in MCUs, then what is the one people call ‘micro tvm runtime’?

Great question. microTVM is the name of the project within TVM to run code on bare-metal platforms. It’s composed of a few different pieces, some of which are shown in that picture above.

• microTVM uses the TVM C runtime, which isn’t directly called out in the picture you sent, though the Graph Runtime mentioned there (now called GraphExecutor) refers to a graph-level model driver built for the C runtime.
• you could produce identical code for the compiled operators with either a microTVM target or a traditional e.g. full-linux TVM targets. that is to say, the difference in the Target is essentially specifying to use the C runtime, and that’s orthogonal to the generated operator impl. now, it’s not possible to actually run the compiled operators on most of the microTVM targets using the C++ runtime used with traditional TVM targets. However, here I’m noting that microTVM shares the same compilation pipeline used by most other applications of TVM.

@sho

Also I’m a bit confused with the difference between tvm and micro tvm. tvm can also generate the same components (Simplified Parameters, Compiled Operators and Graph JSON) right? I’d like to know the difference of the artifacts between by tvm and micro tvm.

hopefully my last answer clarifies this. also note that this is a bit dated–for TVM as a whole, we are building an AOT Executor which requires no JSON parsing. the first application of this is microTVM, and there are tests checked-in to demonstrate this. we’ll be expanding this executor’s capabilities in the coming weeks/months.

@sho

According to the presentation below: TVM Community Meetup - AOT - Google Slides the difference between compilation with AOT and without AOT is the generated C code. To my understanding, AOT enables us to remove Graph Runtime and Graph JSON from the lists stored in MCUs, and instead another kind of C code will be generated which kind of combine compiled operators, Graph Runtime and Graph JSON. All we need to run inference is this C code(Compiled Operators) , Simplified Parameters, Inputs, device initialization code and application code(those in main). We just download these into MCUs. Is my understanding correct?

that’s correct

@sho

After successfully running the script, am I supposed to get all the C code to build and download into MCUs?

before you can run this example, you need to first build TVM. either:

1. follow the microTVM reference VM instructions to build a Virtual Machine which contains all of the dependencies you need to work with microTVM. this is heavyweight, but it is also our reference platform, so it is expected to work out of the box.
2. if you already have Zephyr installed, you don’t need to do this, but per your error you do need to build TVM.

once you’ve built TVM, it will be able to proceed past that error; either path above will work. you should then expect it to build/flash/run code on the micro device.

@sho

Another script for AOT tvm/python/tests/relay/aot/test_crt_aot.py (https://github.com/apache/tvm/blob/main/tests/python/relay/aot/test_crt_aot.py )

this script is intended as a regression test to ensure the AOT remains functional as we develop it. we have some examples as to how to deploy AOT, but they are limited right now. see my next response.

@sho

There is no information on the environments so I didn’t even run the script. But this script generates the C code that can be compiled by let’s say arm v7?

kind of. this script does target Corstone-300, which is a Fixed Virtual Platform intended to help verify Cortex-M profile code. so, it does in fact generate code which could run on microcontrollers, but it doesn’t explicitly target them as the focus for that script is our Continuous Integration.

@sho

At this moment, can we generate C code by AOT, make and customize some application using the generated C code?

to launch code on device, our preferred infrastructure is the Project API. there are a couple integrations so far (arduino and zephyr here). Project API is centered around taking a TVM compiler artifact in Model Library Format and integrating it into a template e.g. IDE project so that it can be built/flashed either by a user or by TVM (to run host-driven inference or autotuning; but you can still flash for standalone purposes depending on how the template project is setup).

for AOT, we have a demo using Project API. for the Zephyr template project, the project_type ProjectOption is used to make this selection; you could try adapting the TFLite tutorial here.

however, a final word about AOT–we haven’t yet pushed on a comprehensive tutorial to demonstrate deployment onto the device due to a couple of in-progress efforts:

• USMP, which will perform static memory planning for all tensors in the graph and therefore dramatically reduce the footprint of models.
• Embedded C runtime interface, which has just nearly landed in full, but we haven’t had time yet to put together a formal demo yet. This is a microcontroller-friendly interface to TVM-compiled code.

At this point we are getting quite close to TVMcon, so I am hoping for better tutorials/examples in Q1 2022. however, I would say that we hope to demonstrate some of this at TVMcon, and I’d suggest looking out for the Arduino tutorial which should cover some of these aspects.