[pre-RFC] TVMC: Add support for microTVM targets

Summary

This RFC obsoletes RFC [0].

This RFC depends on RFC [1].

This pre-RFC aims to discuss the details and collect inputs about how TVMC (TVM CLI tool) can be extended to support microTVM targets, considering the variety of platforms supported by microTVM, like Zephyr and Arduino, and also considering future platforms, taking into account the use of custom/adhoc platforms provided by the users at their convenience.

The interface here proposed relies on the new Project API available on microTVM.

PR 9229 [2] implements the interface in question.

[0] [RFC] TVMC: Add support for µTVM – [RFC] TVMC: Add support for µTVM

[1] [RFC][Project API] Extend metadata in ProjectOption – https://github.com/apache/tvm-rfcs/pull/33

[2] [TVMC] Add new micro context – https://github.com/apache/tvm/pull/9229

Motivation

Currently if a microTVM user wants to compile, build, flash, and run a model for a micro target available on a supported microTVM platform – like Zephyr and Arduino – it’s necessary to drive the process end-to-end via Python scripts. This is ok when creating tests and developing or experimenting with new microTVM code, for example, but for most users that just want to try a new models of interest on a micro target that way can be counterproductive.

TMVC is a command-line driver for TVM (tvmc) that allows users to control the compilation and execution of models on a plethora of targets, using only the CLI, however it currently does not support microTVM targets, so it can not drive a similar workflow (compile, run, etc) on microcontrollers.

This RFC proposes to extend TVMC to support the microTVM targets and platforms so one can easily use tvmc to compile, build (a device image with the compiled model of interest), flash, and run a model on a micro target using only a command-line interface.

Guide-level explanation

Micro targets have unique requirements which justifies the creation of a new context in tvmc called micro to accommodate them.

For instance, a given micro target can exist in more than one supported platform, like the STM32F746 MCU that can be supported by Zephyr and by FreeRTOS. Hence a platform must be selected in addition to the MCU and/or the board.

Moreover, the build and flash steps, although might not be totally exclusive on micro targets, are quite specific (like using the toolchain and the flash tool enabled in the platform), so much it justifies them to exist inside the new micro context, i.e. under tvmc micro.

In that sense, the following subcommands are available under tvmc micro:

1. create-project

2. build

3. flash

create-project purpose is for creating a project directory based on (A) an already compiled model kept in a Model Library Format (MLF) archive, (B) a template associated to a platform, like Zephyr, allowing one to specify several options specific to the selected platform, like the board and extra files to be included in the project directory to be created.

build purpose is for building a firmware image ready to be flashed to the board specified. For example, if Zephyr platform, to build a zephyr.{elf,hex} image. Pertinent options for ‘build’ given the platform selected as an option under ‘build’ will become available.

flash purpose is for effectively flashing the built firmware to the selected board. Again, pertinent options for ‘flash’ given the platform selected will be made available for the user, like for example options regarding the serial port number to be used when flashing the image.

For all these subcommads (1, 2, and 3) the platform template can be selected among the default platforms (zephyr and arduino) or can be a custom one, in that case the platform template must be selected instead. Upon selecting the template platform option -d TEMPLATE_DIR will become available and one can so specify the adhoc template dir (which must contain an adhoc Project API server as per the Project API specification).

A key-value pair is used to specify the options specific to the platforms. If the option is of type ‘bool’ the values available are ‘true’ and ‘false’. Details on project option types are described in RFC-0020.

An example for that interface (using Zephyr) follows:

  1. create-project subcommand:
$ tvmc micro
usage: tvmc micro [-h] {create-project,build,flash} ...
tvmc micro: error: the following arguments are required: subcommand

$ tvmc micro -h
usage: tvmc micro [-h] {create-project,build,flash} ...

optional arguments:
  -h, --help            show this help message and exit

subcommands:
  {create-project,build,flash}
    create-project      create a project template of a given type or given a template dir.
    build               build a project dir, generally creating an image to be flashed, e.g. zephyr.elf.
    flash               flash the built image on a given micro target.

$ tvmc micro create-project
usage: tvmc micro create-project [-h] [-f] PROJECT_DIR MLF {zephyr,arduino,template} ...
tvmc micro create-project: error: the following arguments are required: PROJECT_DIR, MLF, platform

$ tvmc micro create-project /tmp/x100 ./sine_model.tar zephyr
usage: tvmc micro create-project PROJECT_DIR MLF zephyr [--list-options] -o OPTION=VALUE [OPTION=VALUE ...]
tvmc micro create-project PROJECT_DIR MLF zephyr: error: the following arguments are required: -o

$ tvmc micro create-project /tmp/x100 ./sine_model.tar zephyr --list-options
usage: tvmc micro create-project PROJECT_DIR MLF zephyr [--list-options] -o OPTION=VALUE [OPTION=VALUE ...]

optional arguments:
  --list-options        show all options/values for selected platforms/template.
  -o OPTION=VALUE [OPTION=VALUE ...]
                        extra_files_tar=EXTRA_FILES_TAR
                          if given, during generate_project, uncompress the tarball at this path into the project dir.

                        project_type={aot_demo, host_driven}
                          type of project to generate. (required)

                        west_cmd=WEST_CMD
                          path to the west tool. If given, supersedes both the zephyr_base option and ZEPHYR_BASE environment variable.

                        zephyr_board={mimxrt1050_evk, mps2_an521, nrf5340dk_nrf5340_cpuapp,
                                      nucleo_f746zg, nucleo_l4r5zi, qemu_cortex_r5, qemu_riscv32,
                                      qemu_riscv64, qemu_x86, stm32f746g_disco}
                          name of the Zephyr board to build for. (required)

                        config_main_stack_size=CONFIG_MAIN_STACK_SIZE
                          sets CONFIG_MAIN_STACK_SIZE for Zephyr board.

$ tvmc micro create-project /tmp/x200 ./sine_model.tar zephyr -o zephyr_board=stm32f746g_disco project_type=host_driven
$
  1. build subcommand:
$ tvmc micro build
usage: tvmc micro build [-h] [-f] PROJECT_DIR {zephyr,arduino,template} ...
tvmc micro build: error: the following arguments are required: PROJECT_DIR, platform

$ tvmc micro build -h
usage: tvmc micro build [-h] [-f] PROJECT_DIR {zephyr,arduino,template} ...

positional arguments:
  PROJECT_DIR           Project dir to build.

optional arguments:
  -h, --help            show this help message and exit
  -f, --force           Force rebuild.

platforms:
  {zephyr,arduino,template}
                        you must selected a platform from the list. You can pass '-h' for a selected platform to list its options.
    zephyr              select Zephyr platform.
    arduino             select Arduino platform.
    template            select an adhoc template.

$ tvmc micro build /tmp/x200 zephyr
usage: tvmc micro build PROJECT_DIR zephyr [--list-options] -o OPTION=VALUE [OPTION=VALUE ...]
tvmc micro build PROJECT_DIR zephyr: error: the following arguments are required: -o

$ tvmc micro build /tmp/x200 zephyr --list-options
usage: tvmc micro build PROJECT_DIR zephyr [--list-options] -o OPTION=VALUE [OPTION=VALUE ...]

optional arguments:
  --list-options        show all options/values for selected platforms/template.
  -o OPTION=VALUE [OPTION=VALUE ...]
                        verbose={true, false}
                          run build with verbose output.

                        zephyr_base=ZEPHYR_BASE
                          path to the zephyr base directory.

                        zephyr_board={mimxrt1050_evk, mps2_an521, nrf5340dk_nrf5340_cpuapp,
                                      nucleo_f746zg, nucleo_l4r5zi, qemu_cortex_r5, qemu_riscv32,
                                      qemu_riscv64, qemu_x86, stm32f746g_disco}
                          name of the Zephyr board to build for. (required)

$ tvmc micro build /tmp/x200 zephyr -o zephyr_board=stm32f746g_disco
$
  1. flash subcommand:
$ tvmc micro flash
usage: tvmc micro flash [-h] PROJECT_DIR {zephyr,arduino,template} ...
tvmc micro flash: error: the following arguments are required: PROJECT_DIR, platform

$ tvmc micro flash -h
usage: tvmc micro flash [-h] PROJECT_DIR {zephyr,arduino,template} ...

positional arguments:
  PROJECT_DIR           Project dir with a image built.

optional arguments:
  -h, --help            show this help message and exit

platforms:
  {zephyr,arduino,template}
                        you must selected a platform from the list. You can pass '-h' for a selected platform to list its options.
    zephyr              select Zephyr platform.
    arduino             select Arduino platform.
    template            select an adhoc template.

$ tvmc micro flash /tmp/x200 zephyr
usage: tvmc micro flash PROJECT_DIR zephyr [--list-options] -o OPTION=VALUE [OPTION=VALUE ...]
tvmc micro flash PROJECT_DIR zephyr: error: the following arguments are required: -o

$ tvmc micro flash /tmp/x200 zephyr --list-options
usage: tvmc micro flash PROJECT_DIR zephyr [--list-options] -o OPTION=VALUE [OPTION=VALUE ...]

optional arguments:
  --list-options        show all options/values for selected platforms/template.
  -o OPTION=VALUE [OPTION=VALUE ...]
                        zephyr_board={mimxrt1050_evk, mps2_an521, nrf5340dk_nrf5340_cpuapp,
                                      nucleo_f746zg, nucleo_l4r5zi, qemu_cortex_r5, qemu_riscv32,
                                      qemu_riscv64, qemu_x86, stm32f746g_disco}
                          name of the Zephyr board to build for. (required)

$ tvmc micro flash /tmp/x200 zephyr -o zephyr_board=stm32f746g_disco
$

Running a model

As showed above by using tvmc micro subcommands create-project, build, and flash it’s possible to get a model flashed and ready to run on a micro target.

Running a model is naturally common to all targets (not exclusive on MCUs), so commands and options for running a model on microcontrollers are accommodated under the existing tvmc run command. To select a micro target the device ‘micro’ needs to be specified with --device micro. When it happens PATH argument can be used not to selected a compiled model module file (like currently it happens on non-micro targets) but to specify the project directory used for flashing the model of interest:


$ tvmc run --device micro -h
usage: tvmc run [-h] [--device {cpu,cuda,cl,metal,vulkan,rocm,micro}] [--fill-mode {zeros,ones,random}] [-i INPUTS] [-o OUTPUTS] [--print-time] [--print-top N] [--profile] [--repeat N] [--number N]
                [--rpc-key RPC_KEY] [--rpc-tracker RPC_TRACKER]
                PATH

positional arguments:
  PATH                  path to the compiled module file or to the project directory (micro devices only).

optional arguments:
  -h, --help            show this help message and exit
  --device {cpu,cuda,cl,metal,vulkan,rocm,micro}
                        target device to run the compiled module. Defaults to 'cpu'

[...]

Once both micro device and project directory are specified --list-options can be used to list all the options available (optional and required ones) for running the model on the micro target:

$ tvmc run --device micro /tmp/x200 --list-options
usage: tvmc run [-h] [--device {cpu,cuda,cl,metal,vulkan,rocm,micro}] [--fill-mode {zeros,ones,random}] [-i INPUTS] [-o OUTPUTS] [--print-time] [--print-top N] [--profile] [--repeat N] [--number N]
                [--rpc-key RPC_KEY] [--rpc-tracker RPC_TRACKER] [--list-options] --options OPTION=VALUE [OPTION=VALUE ...]
                PATH

positional arguments:
  PATH                  path to the compiled module file or to the project directory (micro devices only).

optional arguments:
  -h, --help            show this help message and exit
  --device {cpu,cuda,cl,metal,vulkan,rocm,micro}
                        target device to run the compiled module. Defaults to 'cpu'
  --fill-mode {zeros,ones,random}
                        fill all input tensors with values. In case --inputs/-i is provided, they will take precedence over --fill-mode. Any remaining inputs will be filled using the chosen fill mode. Defaults to 'random'
  -i INPUTS, --inputs INPUTS
                        path to the .npz input file
  -o OUTPUTS, --outputs OUTPUTS
                        path to the .npz output file
  --print-time          record and print the execution time(s)
  --print-top N         print the top n values and indices of the output tensor
  --profile             generate profiling data from the runtime execution. Using --profile requires the Graph Executor Debug enabled on TVM. Profiling may also have an impact on inference time, making it take longer to be generated.
  --repeat N            run the model n times. Defaults to '1'
  --number N            repeat the run n times. Defaults to '1'
  --rpc-key RPC_KEY     the RPC tracker key of the target device
  --rpc-tracker RPC_TRACKER
                        hostname (required) and port (optional, defaults to 9090) of the RPC tracker, e.g. '192.168.0.100:9999'
  --list-options        show all options/values for selected platforms/template.
  --options OPTION=VALUE [OPTION=VALUE ...]
                        gdbserver_port=GDBSERVER_PORT
                          if given, port number to use when running the local gdbserver.

                        nrfjprog_snr=NRFJPROG_SNR
                          when used with nRF targets, serial # of the attached board to use, from nrfjprog.

                        openocd_serial=OPENOCD_SERIAL
                          when used with OpenOCD targets, serial # of the attached board to use.

                        zephyr_base=ZEPHYR_BASE
                          path to the zephyr base directory.

                        zephyr_board={mimxrt1050_evk, mps2_an521, nrf5340dk_nrf5340_cpuapp,
                                      nucleo_f746zg, nucleo_l4r5zi, qemu_cortex_r5, qemu_riscv32,
                                      qemu_riscv64, qemu_x86, stm32f746g_disco}
                          name of the Zephyr board to build for. (required)

Already existing options that are quite useful when running or testing a model become automatically available when running a model on microcontrollers, like, for instance, --print-top and --fill-mode options.

For example, one can run the flashed example model showed above this way:

$ tvmc run --device micro /tmp/x200 --options zephyr_board=stm32f746g_disco --print-top 1 --fill-mode random
[[ 0.        ]
 [-0.48289177]]
$

tvmc run will take care of opening a transport channel (usually via serial) with the target device where the model is, then it will write the necessary input data to the run the model and collect back the output, showing it on the command-line interface if that’s requested.

Reference-level explanation

The proposed interface relies on Project API server_info_query method to obtain all the options available for a specified platform.

argparse is used to build the command-line parser, but a dynamic approach is used to build a sort of dynamic parser: first one parser is built to parse the platform type; then, once the platform is parsed from the given command line a query is made using the Project API server_info_query method to obtain all the options available for the specified platform and the parser is incremented to parse the options based on the selected platform and subcommand.

Using metadata in ProjectOption associated to every project option returned by server_info_query it’s possible to build the second parser to parse the options by platform and selected subcommand.

The main reason for using that “dynamic parser” mechanism is due to the fact that querying the options for a platform takes about 0.4 seconds. So with the 2 default platforms current available (Zephyr and Arduino) it takes about 1 second to query the options and build a parser accordingly. Thus with 3 or more default platforms the user would start to feel some pain when using tvmc.

The current implementation avoids querying the options for all the default platforms available and so will always spend ~0.4s to query and build the proper parsers to parse the platform options no matter how many platforms might exist.

Default project templates

Default project templates must be detected and made available as platform options automatically.

The mechanism for automatically detecting the default template directories must consider basically two different environments: when TVM is built from sources and when TVM is installed via a Python package (like TVM tlcpack).

The mechanism here suggested is one similar to the one used by tvm.micro.get_standalone_crt_dir(), which by its turn relies on _ffi.libinfo.find_lib_path() and so can correctly find the libraries taking into account if TVM being used is from sources or from a installed package.

Hence for the specific case of the template directories a new function named tvm.micro.get_template_dirs() which returns all available default platform templates directories will be created. The templates themselves will be added to the wheel package accordingly, just like the standalone_crt/ is added.

Drawbacks

argparse unfortunately only supports two mutually exclusive groups through add_mutually_exclusive_group(). This complicates a smooth integration of new micro target specific options into the already existing tvmc run command.

For example, --rpc-key and --rpc-tracker do not apply to micro targets, but can’t be automatically excluded by argparse if micro target specific option --list-options is used. This maybe can be solved by enforcing the mutual exclusive groups in the code instead of in the parser as it’s done in the current implementation and mark these options with something like (not available for micro targets), just as required options are marked with (required).

Unresolved questions

  1. Currently required options that are passed when creating a project, i.e. passed to create-project, need to be repeated for build, flash, and run. This is ok if one, for whatever reason, wants to change it from one subcommand to another, but might be a burden to the user that will need to type the key-value again for each subsequent subcommand. Maybe the required values passed when create-project is used could be recorded in the new project directory (in a JSON file – .tvmc.json) and a flag called --auto can be added to the subcommands (build, flash, and run`) and if the required options where already passed and are recorded in the JSON file then TVMC would pick up and use them automatically.

  2. Similarly it’s possible to remove the platform selection for other subcommands after the platform is specified once in create-project. In that case the project options would be discovered automatically based on the project dir, not based on the template dir, as it’s done in the current implementation.

Future possibilities

  1. The current implementation and RFC does not address tvmc tune for micro targets. It will be addressed later, once the interface here proposed is discussed and crystallizes. The ‘tvmc tune’ will be extended probably along the lines as the tvmc run command is extended.

  2. For creating a project one departs from a compiled model kept in a Model Library Format archive. This is fine and gives to the user total control about which settings would be used when compiling a model (using tvmc compile). On the other hand the CPU model for the target board, for instance, needs to be specified explicitly, and that maybe annoying for some users. Thus it would be good in the future if create-project could depart from a model not compiled and deduce a default working set of options to compile the model based on the specified board.

2 Likes

cc @leandron @areusch @manupa-arm @tgall_foo @Mousius @mehrdadh

Thanks @gromero for the RFC. Overall looks good to me. I have few comments/suggestions:

  • considering the flow of using tvmc for micro, I find it’s a little bit confusing to switch between tvmc and tvmc micro back and forth. My suggestion is, if possible, to have all tasks for microTVM tvmc to live under tvmc micro including [tvmc micro compile, tvmc micro run, etc] and then reuse tvmc interface in the backend. This will also resolve the drawback that you mentioned, because you can limit arguments for tvmc micro run and pass it to tvmc run.

I think this is a great idea for next version. I suggest that we keep compile option, but change create-project to compile and create the project.

Another suggestion that I have is renaming create-project to create. project in this context doesn’t really differentiate this subcommand to others.

@mehrdadh Thanks a lot for the review!

  • considering the flow of using tvmc for micro, I find it’s a little bit confusing to switch between tvmc and tvmc micro back and forth. My suggestion is, if possible, to have all tasks for microTVM tvmc to live under tvmc micro including [tvmc micro compile, tvmc micro run, etc] and then reuse tvmc interface in the backend. This will also resolve the drawback that you mentioned, because you can limit arguments for tvmc micro run and pass it to tvmc run.

As a general “rule” I’ve tried to stick with not duplicating existing commands when adding the new micro context.

Regarding the compile command, besides avoiding duplicated commands the reasons to separate it from micro build and not creating a compile command inside the “micro” context are also along the lines of what was previously discussed with @leandron , @manupa-arm , and @areusch in [0] – comment #7 I think it’s being working fine that split so far.

Regarding the run command, although creating a micro run would avoid the drawback mentioned it would also duplicate all the options that are common to the non-micro targets inside micro run. Also the more we go this way the more tvmc micro will look like “a tvmc (micro) inside tvmc”, like an whole new tool for micro targets only. So there are caveats with that approach too. But I’m happy to debate more about it.

I think this is a great idea for next version. I suggest that we keep compile option, but change create-project to compile and create the project.

If the compile command you mention here is the one that already exists (i.e. tvmc compile) I agreed! tvmc micro create-project can perfectly accommodate your suggestion - compile and create the project, yep.

1 Like

Sure. I have no strong feelings about it. If nobody objects, I’m happy to change it in v2. Thanks.

1 Like

@gromero thanks for writing this up! broadly looks good, a couple points to ask about:

In that sense, the following subcommands are available under tvmc micro:

1. create-project

2. build

3. flash

it might be worth just stating here that build and flash are intended to help with debugging inference flows built on top of this infrastructure. they’re not intended to replace the user’s workflow (e.g. users can still build and flash independently of this).

The main reason for using that “dynamic parser” mechanism is due to the fact that querying the options for a platform takes about 0.4 seconds.

I’d also say that it’s possible for the user to supply their own platform, so it’s not possible to enumerate all platforms’ commands in advance.

This maybe can be solved by enforcing the mutual exclusive groups in the code instead of in the parser as it’s done in the current implementation and mark these options with something like (not available for micro targets) , just as required options are marked with (required) .

seems reasonable.

could you explain this one a bit more?

i suggest we keep create-project in case we ever need to create something else. we may even consider e.g. generate-project instead. however, it is a bit wordy if we do expect people to type this a lot…in this case if we decide we should shorten it, i’d propose we keep the longer version and introduce an alias.

1 Like

@areusch Thanks a lot for the review Andrew!

oh indeed, that is important. I’ll improve it. Thanks!

Fair enough. I’ll put that reason first.

Sure. I meant that currently one does:

$ tvmc compile ./micro_speech.tflite --target="c -keys=cpu -link-params=0 -march=armv7e-m -mcpu=cortex-m7 -model=stm32f746xx -runtime=c -system-lib=1" --output micro_speech.tar --output-format mlf --pass-config tir.disable_vectorize=1 --disabled-pass="AlterOpLayout"`

and then:

tvmc micro create-project /tmp/micro_speech ./micro_speech.tar zephyr -o project_type=host_driven zephyr_board=stm32f746g_disco

But since in 2. the board is given (zephyr_board=stm32f746g_disco) a default value for that board can be used for options --target, --pass-config, and --disabled-pass in 1. so tvmc micro create-project can be enhanced to also take care of compiling the model, so step 1. can be skipped. Of course, tvmc compile will always be available if one would like to take full control of the these options when compiling a model.

I chose “create” instead of “generate” indeed because I thought of “generate” being wordy. But honestly I think “create” is just a little better in that sense. I’ll keep create-project for now. Of course I can use an alias cp for it – or create. @mehrdadh If you were also concerned about the boredom of typing -project adding the alias might also help partially – if there are no objections.

As a reference it would look something like:

gromero@amd:~/git/tvm/python/tvm/driver/tvmc$ tvmc micro -h 
usage: tvmc micro [-h] {create-project,create,cp,build,flash} ...

optional arguments:
  -h, --help            show this help message and exit

subcommands:
  {create-project,create,cp,build,flash}
    create-project (create, cp)
                        create a project template of a given type or given a template dir.
    build               build a project dir, generally creating an image to be flashed, e.g. zephyr.elf.
    flash               flash the built image on a given micro target.
1 Like

I like the idea of reusing previous arguments and I think JSON file could work.

1 Like

I think using alias is perfect, but this looks weird:

Blockquote subcommands: {create-project,create,cp,build,flash}

is there a way to make it like this:

gromero@amd:~/git/tvm/python/tvm/driver/tvmc$ tvmc micro -h 
usage: tvmc micro [-h] {create-project,build,flash} ...

optional arguments:
  -h, --help            show this help message and exit

subcommands:
  {create-project,build,flash}
    create-project (create, cp)
                        create a project template of a given type or given a template dir.
    build               build a project dir, generally creating an image to be flashed, e.g. zephyr.elf.
    flash               flash the built image on a given micro target.

I thought that too at first, but honestly on a second thought that’s perfectly correct. Think about (create, cp) text there but without that information in the list of subcommands, to me it would be a bit tricky to find out that it means that it is an alias for create-project subcommand. Either way, to the best of my knowledge, there is not way to change it.

ah i understand. yeah this makes sense to me. it might help to clarify that by “departs from a compiled model” you mean “may supply a model in one of the frameworks’ format (e.g. TFLite, ONNX, etc), and therefore may need to also supply a target string.” But also, you do have this problem when running compile as well, just there is no way to lookup the target string from Project API. i’m not suggesting we necessarily need to add that (can always do that in a v2), but just saying it’s merely that you’re including one step inside another.

i think the formatting is ok, but we should choose something else than cp since cp means copy to many linux folks.

1 Like

By “compiled model” in the text I meant the one already compiled and kept in a MLF archive… did you mean “non-compiled model”? :slight_smile: Either way I agree it’s better to cite the input formats like TFLite, ONNX, etc to avoid confusion when I say "[...] could depart from a **model not compiled** and deduce a default working set of options [...]" :+1: I’ll change it. Thanks.

yeah I was actually thinking of that as a duty for tvmc only. It would be tvmc that would keep a table associating boards vs default target options, not necessarily for all boards.

Yep, I was thinking of it more for a v2. :+1: