[RFC] TVMC: Add support for µTVM

** Introduction **

This RFC aims to detail the integration of microTVM into TVMC and the tvmc commands their options available when working with targeting devices supported by microTVM.

Currently in microTVM the entire compile/flash/run/debug process is driven end-to-end by the user, via microTVM Python API modules, as it happens in ‘tutorials/micro/micro_tflite.py’ script.

TMVC is a command-line driver for TVM (tvmc) that allows users to better control the compilation, execution, and tuning of several input models on various hardware targets. However it currently doesn’t support microTVM targets, so it can not drive a similar workflow (compile, run, tune) on targets supported by microTVM, like a variety of microcontrollers.

Endowing tmvc to support the mentioned workflow stages for microTVM targets allows developers to focus on testing, fixing bugs, and improving each stage independently, so not need to drive the whole process end-to-end anymore. Moreover, the workflow proposed by this RFC roughly corresponds to the common steps in the average developer workflow: compile a model, flash it (new stage proposed), run it, and tune.

** Propositions **

To achieve the goal of integrating microTVM to TVMC (tvmc tool), the following changes are proposed:

1- New command

A new command, ‘flash’, must be added to tvmcin order to control flashing to the microTVM devices. This adds a new stage in the existing TVMC workflow (compile, run, tune).

2- “Glue” between stages

In order to coordinate between the different stages in the workflow (compile, flash, run, tune) it’s necessary to have a persistent file with metadata describing several aspects of the build (like the SDK used to generate the binary image, the target MCU and the board) and an archive (.tar) containing at least:

a) the generated binary image to be flashed (like zephyr.{elf,hex}); b) the graph (mod.json); and c) model parameters (mod.params).

Artifacts a, b, and c are all generated on the compile stage. The artifact a is used then on the flash stage and, finally, b and c artifacts are used on the run stage only.

It’s also proposed that adding information about the target and board used in the compile stage into the metadata file avoids the need to specify the ‘–device’ flag again on next stages, like for ‘run’, since the target and board were already specified previously and can’t change…

Finally, a mechanism should exist to avoid running a model on a target not prepared for it. Hence a signature is proposed to avoid the case when one tries to run a model against a target that was not previously flashed with the correct binary image supporting the necessary operations to run the model. The signature would be generated and stored on compile time as a static member of the RPC server and a simple RPC call whould be available to enable requesting the signature. A copy of the signature would be kept in a metadata file and tvmc run command would request the target’s signature via a RPC and compare it against the copy saved in the metadata file. If the signatures match, then tvmc runs the model, otherwise it aborts the run process and informs the user about the mismatch.

3- Selecting microTVM targets

There are currently two propositions to enable microTVM targets:

  1. Use flag --micro (see prototype below), which will make available additional flags to select the board (–board) and the SDK used to generate the image (–sdk). That option is quite easy to implement but it turns out to result in a kind of “fork” inside tvmc, using not much of the common code currently used for TVM. Example:
$ python3 -m tvm.driver.tvmc compile --micro --targe=stm32f746xx --board=stm32f746g_disco --sdk=zephyr sine_model.tflite
  1. Build upon PR #7304 [1] and expand --target. The benefits in theory would be that it would allow for a better use of the existing code. Example:
$ python3 -m tvm.driver.tvmc compile --target="zephyr -targe=stm32f746xx -board=stm32f746g_disco" sine_model.tflite

** Prototype **

A prototype leveraging the microTVM API, the same API presented and used by the “micro_tflite.py” script for the micro tutorial is provided as a reference [0]. A typical workflow would be like [2].

The prototype splits the run process in two stages : flash (introducing a new ‘tvmc’ command ‘flash’ for that goal) and run, so when one runs a model it’s not necessary to flash the binary image again, avoiding target flash memory wear and, in some cases, easing the debug flow. For example, on some ARM MCUs a ST-Link is used both for flashing the device and for providing a GDB serve abstraction for debugging, so it’s not possible to attach to the device via GDB and at the same time flash the device, so it’s tricky to debug microTVM using micro_tflite.py script or a similar one because running is tied to flashing (end-to-end) (tvm.micro.Session(binary=micro_binary, flasher=flasher). The prototype eases the debugging process because it’s possible to only run the model without flashing a new image automatically.

The prototype doesn’t address the Project API as described in [3]. It also doesn’t implement the auto tuning (‘tune’ command for microTVM targets).

[0] GitHub - gromero/tvm at tvmc

[1] [TVMC] Add custom codegen (BYOC) passes for compilation and tuning by leandron · Pull Request #7304 · apache/tvm · GitHub

[2] https://people.linaro.org/~gustavo.romero/tvm/tmvc_microtvm_prototype.txt

[3] [µTVM] Mini-Map: the µTVM Developer Workflow

@tqchen @thierry @mdw-octoml @ramana-arm @manupa-arm @leandron @tgall_foo @liangfu @areusch

Hi @gromero,

Thanks for the proposal.

I have a concern regarding the tvmc compile and how it should it be unified/seperated with existing flows for non-micro flows.

My view is we should be primarily be able to provide a path way to get something proposed/(eventually get agreed) as the Model Library ([µTVM] Mini-Map: the µTVM Developer Workflow)

I think firmware compilation should be seperate from tvmc compile, however its upto to debate whether we want a seperate peer to compile subcommand or an intergrated option for the compile subcommand. I think most users would want to use TVM directly embedded in the application (w/o going through a RPC server) – similiar what’s done here : ([µTVM] Static Runtime Code Generator - #2 by areusch). Thus, we would not want to tvm to create the final .hex/.elf in tvmc compile but maybe we can have tvmc compile --firmware / tvmc <someother command to indicate firmware compilation>.

I think if we agree tvmc compile is to be reserved for the creation of Model Library, then we can have alignment/unification with current tvmc target specification. What do you think?

Looking at the overall flow, and to which public it is aimed, I think most of the work done here can be reused, but needs to be decoupled into separate components, in the way described by @areusch in ([µTVM] Mini-Map: the µTVM Developer Workflow).

In that way, we would leave the door open for integration with other RTOS flavours, as well as keeping the separation of concerns to bridge TVM with more than only Zephyr.

From that perspective, tvmc continues providing an output that we can then plug into an embedded project, to be used.

I think would be very valuable for the end-user experience, and can be done despite all discussed here in a separate PR.

I suggest we reach an agreement on [µTVM] Mini-Map: the µTVM Developer Workflow, and then revisit the strategy here to adjust it accordingly.

cc @mjs

Hi Manupa,

Thanks a lot for the review!

My view is we should be primarily be able to provide a path way to get something proposed/(eventually get agreed) as the Model Library ([µTVM] Mini-Map: the µTVM Developer Workflow)

Yes, I agree. My proposal conflicts with what is proposed by Andrew on the Mini-Map regarding the Model Library and the Project-level API. I’ll try to update my proposal to reflect it.

I think firmware compilation should be seperate from tvmc compile, however its upto to debate whether we want a seperate peer to compile subcommand or an intergrated option for the compile subcommand. I think most users would want to use TVM directly embedded in the application (w/o going through a RPC server) – similiar what’s done here : ([µTVM] Static Runtime Code Generator - #2 by areusch). Thus, we would not want to tvm to create the final .hex/.elf in tvmc compile but maybe we can have tvmc compile --firmware / tvmc .

I understand that currently tvmc already generates a .tar containing a mod.so object and that shared object can then be either passed to a remote RPC server to be executed or be loaded by the application directly (embedded in it). tvmc generates that .so object on the ‘compile’ stage and that won’t change for Library Module and it will reside in lib/ folder. So for the microTVM targets, that will use the Project API, I think it also makes sense to generated an object ready to be run/loaded on the ‘compile’ stage. How about ‘compile’ command build the relay model, generated the lib0,1,n.c and devc.c (the C sources) – as it does already – and then trigger the firmware build via the Project API that, by its turn, will finish producing the .bin, .hex, .elf files? I think the firmware image in the microTVM has a similar “role” as the .so to be used via a RPC call or when embedded in a application. I’ll comment it too on the Mini-Map topic.

One thing that occurred to me is that we haven’t specify for the Project API where the the parameters.json and the graph,json will reside. Maybe in a bundle inside each project?

I think if we agree tvmc compile is to be reserved for the creation of Model Library, then we can have alignment/unification with current tvmc target specification. What do you think?

I can’t see why tvmc ‘compile’ stage needs to be reserved only for the creation of the Model Library case. Why can’t it also drive the Project-level API case? Well, one thing is that I think that with the new design proposed in the Mini-Map only the target has to be specified, not the board (–board), because that information will be in the project metadata. Then tvmc would pass only the specified target to the SDK used by the project (like Zephyr’s SDK). But maybe you’ve spotted a case I can’t see. Would you mind to explain to me a tad further?

Hi Leandro,

Thanks a lot for the review!

I think would be very valuable for the end-user experience, and can be done despite all discussed here in a separate PR.

Got it. I’ll do it.

I suggest we reach an agreement on [µTVM] Mini-Map: the µTVM Developer Workflow , and then revisit the strategy here to adjust it accordingly.

Yes, I agree :slight_smile:

Hi @gromero ,

I think the firmware image in the microTVM has a similar “role” as the .so to be used via a RPC call or when embedded in a application. I’ll comment it too on the Mini-Map topic.

So the RPC feels like an optional layer above the runtime that calls into the runtime. When you have an fully fledged OS we can rely on dlopen to perform the necessary linking, thus it was not an issue in the non-micro TVM use cases.

So for tvm micro use-cases that does not want a RPC in the middle of the application, following is the approach we are currently pursuing. Feel free to suggest if there is something else that is better.

In this use case, we in the run we dont have an RPC component at all and the makefile of the bundle_static is the one generating the final .elf/.hex/.bin incoporating sources/binaries/static archive built by tvm. Let’s just say tvm produces the sources as you say (lib0,1,n.c and devc.c) for now. I think Model Library is proposed as .tar to encapture these :

metadata.json - Overall metadata describing this build
    TVM target
    Description/hash of original model
    Original parameters?
    Other state needed from model compilation later in the pipeline
crt/ - The content of standalone_crt from TVM build/
lib/ - Stores generated binary libraries
parameters.json - JSON-serialized Simplified Parameters (or this could be binary format)
README.md - Perhaps a short standardized README for new users
runtime-config/ - Stores runtime configuration.
    For GraphRuntime, graph.json should be created in this directory.
src/ - Stores generated C source

Therefore, the firmware image sounds like tvm is doing something additional that makes the RPC server mandatory for deployment. Please correct me here if I am wrong. In comparison to non-micro flows which builds .so, we could validly still use the .so in the presense of an OS for an application like bundle static.

One thing that occurred to me is that we haven’t specify for the Project API where the the parameters.json and the graph,json will reside. Maybe in a bundle inside each project?

There is a place inside the proposed Model Library. Maybe we can use that.

I can’t see why tvmc ‘compile’ stage needs to be reserved only for the creation of the Model Library case. Why can’t it also drive the Project-level API case?

Maybe “reserved” is not the correct word here :slight_smile: . For the reasons above, I feel the vanilla tvmc compile should produce Model Library and if we mention something extra (e.g., tvmc compile --firmware – Im not too sure about the exact command) we could do the additional step of making it a firmware image bundled with the RPC ?. If the general consensus is that this should be flipped as in (tvmc compile should by default give the firmware image and tvmc compile --model_lib should give the Model library) that could work as well. However, my point is we should enable a Model Library pathway as that is the primary artifact that is produced by tvm that is naturally compiled for the model given.

hi @gustavo,

Thanks for posting this up and apologies for the late review! I think tvmc integration would be a very positive thing for µTVM. Here are some thoughts:

→ First off, I agree with @manupa-arm and @leandron that we should split the compilation process into two pieces:

  1. performing tvm.relay.build on a model (e.g. tvmc compile)
  2. compiling a firmware binary (e.g. tvmc micro build)

I think making this split gives us a few advantages, the main one being that it allows tvmc to be a tool in development workflows we haven’t yet envisioned. This also fits well with the way the rest of TVM is built.

→ Model Library Format

2- “Glue” between stages

In order to coordinate between the different stages in the workflow (compile, flash, run, tune) it’s necessary to have a persistent file with metadata describing several aspects of the build

I think this sounds very similar to Model Library Format, so I think we have similar concerns. If we make the split above, it seems like they serve a similar purpose. I think it would be great to wrap any concerns here into the Model Library Format RFC, which we should write soon.

→ Commands

I think it might be interesting to contemplate making a micro sub-command of tvmc for build, flash, and perhaps run. This would provide a place for micro-specific commands without cluttering the top-level command tree of tvmc, which could confuse non-micro users. So this would be:

  1. tvmc compile
  2. tvmc micro build
  3. tvmc micro flash
  4. tvmc run

These commands should line up well with the Project API–the main difference being that commands #2 and #3 would be handled entirely by the generated project’s build system, and a new command tvmc micro gen-project (or something) would be added to generate the e.g. Zephyr project.

It would also be great if, for now, the tvmc micro subcommands all took an explicit command-line arg to that identifies the project using the directory where the project lives (or should live for tvmc micro build). ZephyrCompiler needs this information anyway–it just doesn’t treat it as a “template project” for now. I think it would also be fine in the tvmc micro build implementation to shutil.copytree the template project (for now, given as a separate e.g. --template-project arg) into the location specified on the command-line.

→ Selecting microTVM targets

I think if we specify --target on the command-line, that should be the TVM target string. We could add “zephyr” to that target string, but here is one thing to consider: the point of that target string is to capture, for autotuning’s sake, the configuration of the code and target device up to the point it impacts the timed implementation. For this, I’d propose that an e.g. git sha1 of the Project API repo should be enough.

After the Project API refactor, we will need to identify to tvmc micro gen-project which Project API to use. This can be done through the project paths given on the command line, assuming we take the approach in the mini-map of placing the project API implementation in a known file e.g. $project_path/microtvm.py. So, I’m not sure we need to explicitly mention Zephyr by name in the tvmc command line.

→ Misc thoughts

It’s also proposed that adding information about the target and board used in the compile stage into the metadata file avoids the need to specify the ‘–device’ flag again on next stages, like for ‘run’, since the target and board were already specified previously and can’t change…

I agree with this but I would also propose the caveat that device serial numbers be left out of the generated metadata. The reason is that these are specific to the machine while compiled firmware may not be, so placing them in the Model Library Format metadata and then downstream artifacts may restrict portability.

→ Next steps

I think it would be great to agree on the Model Library Format and Mini-Map before we proceed too far towards merging this one. Beyond that, I think it should be more-or-less compatible before and after the Project API change. In my mind, I’d thought to do these two in sequence, but I’m glad we have discussed this because I think tvmc support will be a great addition to µTVM and I think we can proceed in parallel.

Let me know your thoughts!

Hi Manupa,

So the RPC feels like an optional layer above the runtime that calls into the runtime. When you have an fully fledged OS we can rely on dlopen to perform the necessary linking, thus it was not an issue in the non-micro TVM use cases.

So for tvm micro use-cases that does not want a RPC in the middle of the application, following is the approach we are currently pursuing. Feel free to suggest if there is something else that is better.

Got it. Thanks for the example provided. I was missing that use-case.

So, for use-cases like in that example, I’m wondering if we should support anything special on the tvmc side. The only thing I see specific to that use-case is that it has an runtime “adapter” for TVM C and C++ interfaces (in bundle.c and bundle.cc) that will be used by the application and will be linked by an ad hoc (per project or application that wants to build the static runtime without a RPC server) against the model_c.o or model_cpp.o objects generated via the TVM build.relay API. Currently tvmc uses runtime.Module.export_library when generating .tar archive, but for that use-case I believe it should provide a .o instead, i.e. use runtime.Module.save (like in the example script)? If that is true I think that this document should address at least a way to specify a .so or .o (as required in that use-case). Then Makefile will call tvmc accordingly just passing the model (for instance, mobilenet0.25) + a flags specifying that a .o must be generated and tvmc take care of generating “.o” to be used.

Now, how about the “adapters” (bundle.c and bundle.cc), it seems to me that they are just boilerplate code that we could put into a template dir just like the template dir for a Project-API and copy from it accordingly when generating the Module Library, as @areusch suggested?

In this use case, we in the run we dont have an RPC component at all and the makefile of the bundle_static is the one generating the final .elf/.hex/.bin incoporating sources/binaries/static archive built by tvm. Let’s just say tvm produces the sources as you say (lib0,1,n.c and devc.c) for now. I think Model Library is proposed as .tar to encapture these.

Yeah I agree Model Library should encapsulate the .c and .cpp and other source files (btw I think this name is a bit misleading, like it takes the name “Library” in it and has more than just libraries inside its guts… but I’ll discuss that in the other RFC related to it). So in the very same example (apps/bundle_deploy), lilb0-n.c, dev.c and other sources would land into src/ and model_c.o and model_cpp.o would land into lib/? Did I get you right?

There is a place inside the proposed Model Library. Maybe we can use that. Yep, I think that too, but would that bundle be inside the project (always) or the working dir where tvmc was executed (by default)?

However, my point is we should enable a Model Library pathway as that is the primary artifact that is produced by tvm that is naturally compiled for the model given.

Yep, definitely.

Hi Andrew!

Thanks a lot for the review.

→ First off, I agree with @manupa-arm and @leandron that we should split the compilation process into two pieces:

  1. performing tvm.relay.build on a model (e.g. tvmc compile)
  2. compiling a firmware binary (e.g. tvmc micro build)

I think making this split gives us a few advantages, the main one being that it allows tvmc to be a tool in development workflows we haven’t yet envisioned. This also fits well with the way the rest of TVM is built.

OK.

→ Model Library Format I think this sounds very similar to Model Library Format, so I think we have similar concerns. If we make the split above, it seems like they serve a similar purpose. I think it would be great to wrap any concerns here into the Model Library Format RFC, which we should write soon.

Right, I think there is a consensus round the need :slight_smile: Model Library Format looks promising, I think it would be good to list all the use-cases (TVM, microTVM using an SDK, static runtime, application dlopen’ning directly a module .so) and examples for each on where (which folder) each kind of files will finally land after each command. I’ll try to make it and comment on the Mini-map.

→ Commands I think it might be interesting to contemplate making a micro sub-command of tvmc for build, flash, and perhaps run. This would provide a place for micro-specific commands without cluttering the top-level command tree of tvmc, which could confuse non-micro users. So this would be:

Yeah, I thought a bit more about it. It sounds good, yep. I think run would need to have micro sub-command :+1:

and a new command tvmc micro gen-project (or something) would be added to generate the e.g. Zephyr project.

oh that’s cool. Copying (or generating) it from the template dir you mentioned right?

→ Selecting microTVM targets I think if we specify --target on the command-line, that should be the TVM target string. We could add “zephyr” to that target string, but here is one thing to consider: the point of that target string is to capture, for autotuning’s sake, the configuration of the code and target device up to the point it impacts the timed implementation.

Right, I think it is not indeed necessary to pass the SDK (like zephyr) to the compile command because I understand build.relay doesn’t need to know anything about the “board” (contrary to what is used in the prototype of this RFC document) and about the SDK used. tvmc micro build maybe should address it?

For this, I’d propose that an e.g. git sha1 of the Project API repo should be enough.

Sorry, I could not follow you here.

I agree with this but I would also propose the caveat that device serial numbers be left out of the generated metadata. The reason is that these are specific to the machine while compiled firmware may not be, so placing them in the Model Library Format metadata and then downstream artifacts may restrict portability.

By “device serial numbers” do you mean the usb serial device number? If so currently we allow only one of such a device to be available and it’s probed, failing if more than 1 serial port is available no? Maybe we should stick with that default and add an option to the user to specify it when desired. Anyway, I think that info should not be in the metadata, it should be left by the user to specify when tvmc micro flash and tvmc run is executed?

Great, let’s discuss Model Library Format a bit further on another RFC. I will try to start one early this week.

and a new command tvmc micro gen-project (or something) would be added to generate the e.g. Zephyr project.

oh that’s cool. Copying (or generating) it from the template dir you mentioned right?

Yes. However, to refine this a bit and address a comment you made earlier to @manupa-arm: this template project shouldn’t get used if we are just producing Model Library Format. The idea of the template project is to provide the necessary boilerplate/glue to configure an SoC to run a model given one in Model Library Format. The template project is tied to the implementation of the Project-level API.

Right, I think it is not indeed necessary to pass the SDK (like zephyr ) to the compile command because I understand build.relay doesn’t need to know anything about the “board” (contrary to what is used in the prototype of this RFC document) and about the SDK used. tvmc micro build maybe should address it?

I think that sounds ok at a high level.

For this, I’d propose that an e.g. git sha1 of the Project API repo should be enough.

Sorry, I could not follow you here.

Here I mean–if we are going to include something to describe the device configuration (similar to the --target=zephyr -target=stm32f746xx -board=... you proposed), the portion which describes the project template used should just be the git sha1 from the git repo that houses the template project and the Project-level API implementation.

By “device serial numbers” do you mean the usb serial device number? If so currently we allow only one of such a device to be available and it’s probed, failing if more than 1 serial port is available no? Maybe we should stick with that default and add an option to the user to specify it when desired. Anyway, I think that info should not be in the metadata, it should be left by the user to specify when tvmc micro flash and tvmc run is executed?

Yeah that’s right, but you could override or choose a specific port when multiple identical boards are connected. I agree it should not be in metadata and instead passed to flash or run either explicitly on the command-line or in a .gitignore'd environment config file.

Great, let’s discuss Model Library Format a bit further on another RFC. I will try to start one early this week.

Cool. Thanks :slight_smile:

and a new command tvmc micro gen-project (or something) would be added to generate the e.g. Zephyr project.

oh that’s cool. Copying (or generating) it from the template dir you mentioned right?

Yes. However, to refine this a bit and address a comment you made earlier to @manupa-arm: this template project shouldn’t get used if we are just producing Model Library Format. The idea of the template project is to provide the necessary boilerplate/glue to configure an SoC to run a model given one in Model Library Format. The template project is tied to the implementation of the Project-level API.

Right, I got that tvmc micro gen-project is just for creating (for instance) a project template for a given microTVM project, like one that will use Zephyr’s SDK, so no need to use for the other use-cases that use Model Library Format (and that don’t use the Project-API).

On my comment to @manupa-arm I was just wondering if it would make sense to use a template (a different one from project template) for use-cases like the static runtime in apps/bundle_deploy, since in my understanding the code in bundle.c (and bundle.cc) is a just a boilerplate/glue that will be used in similar use-cases. In that case a compile flag would exist specifying a static runtime (tvmc currently only generates a .so afaics) (like, --static-runtime) and the template would be copied to Model Library Format into src/ so the user could later easily use it in conjunction with other sources generated by build.relay on a Makefile. The advantage in my view, if that’s possible, would be to avoid repeating it boilerplate on each application that wants to use a static runtime.

Here I mean–if we are going to include something to describe the device configuration (similar to the --target=zephyr -target=stm32f746xx -board=... you proposed), the portion which describes the project template used should just be the git sha1 from the git repo that houses the template project and the Project-level API implementation.

Got it. OK.

Btw, I’m still struggling to visualize where the Model Library Format (MLF) file will reside by default for the various use-cases. By default I’m thinking of it residing in the current working dir when the Project-API is not used and inside the target project when the Project-API is used.

Hi @gromero and @areusch ,

Interesting discussions!

The only thing I see specific to that use-case is that it has an runtime “adapter” for TVM C and C++ interfaces (in bundle.c and bundle.cc) that will be used by the application and will be linked by an ad hoc (per project or application that wants to build the static runtime without a RPC server) against the model_c.o or model_cpp.o objects generated via the TVM build.relay API. Currently tvmc uses runtime.Module.export_library when generating .tar archive, but for that use-case I believe it should provide a .o instead , i.e. use runtime.Module.save (like in the example script)? If that is true I think that this document should address at least a way to specify a .so or .o (as required in that use-case). Then Makefile will call tvmc accordingly just passing the model (for instance, mobilenet0.25 ) + a flags specifying that a .o must be generated and tvmc take care of generating “.o” to be used.

Yes, you are mostly right!. I have also made a similar comment in the other RFC that we should additionally include a compiled artifact .a ( or it could be .o – don’t have a strong preference). I was hinting that we could do use export_library on the runtime.Module (which is a multi-module hierarchical tree) produced via relay.build using the correct cross compiler and target object format as .o/.a. However, I agree with @areusch that we still need the sources as Zephyr (west) and similar flows accept the direct sources to create alignment with compilation flags across all sources compiled to create final .elf/.bin/.hex.

On my comment to @manupa-arm I was just wondering if it would make sense to use a template (a different one from project template) for use-cases like the static runtime in apps/bundle_deploy , since in my understanding the code in bundle.c (and bundle.cc ) is a just a boilerplate/glue that will be used in similar use-cases. In that case a compile flag would exist specifying a static runtime (tvmc currently only generates a .so afaics) (like, --static-runtime) and the template would be copied to Model Library Format into src/ so the user could later easily use it in conjunction with other sources generated by build.relay on a Makefile. The advantage in my view, if that’s possible, would be to avoid repeating it boilerplate on each application that wants to use a static runtime.

I think its OK to bundle them in the model library as optional/helper sources but I think we should not link them in the compiled artifact (.o/.a). Yes, we would need a tvmc compile flag to specify the output type. I think we should discuss the contents of Model Library in that RFC

Catching up with all the messages here, it looks like there are two problems and two solutions for the subsets of tvmc commands in discussion:

P1: How can we create something that allows uTVM to be integrated bare-metal and RTOS? This is aimed to provide TVM/TVMC as tools to be integrated as part of an embedded process. The target user knows about their project (how to build it in the most optimised way, how to flash it on a board, how to debug, etc.) and will fit something we provide in the Model Library Format as part of their workflows.

  • S1: This is solved by using the improving tvmc compile to generate a more comprehensive set of outputs in the Model Library Format, so that users can pick what is relevant for them (header files, static object files, boilerplate glue, etc). tvmc run has no role in this “uTVM” use case.

P2: How to test/tune operators and models using TVM/TVMC, when aiming at embedded devices? This relates to TVM developers, that not necessarily know a lot about embedded development, but want to validate and check performance using embedded targets.

  • S2: In this case, we would have the proposed tvmc micro namespace, which would be basically aimed for testing and to bridge TVM with already existing tools, provided by each RTOS flavour the community decides to integrate with. This also would need to enhance tvmc tune, so that we could support uTVM tuning on this workflow.

With all that, I just wanted to check we are talking about the same end-users when we say users, and to seeing the problems we are solving them with new commands, and who are the commands aimed at.

Do you see any other fundamental problems we are trying to solve with all being discussed in this thread, that is not captured here?

@manupa-arm @leandron thanks for the reply!

@manupa-arm wrote:

I was hinting that we could do use export_library on the runtime.Module (which is a multi-module hierarchical tree) produced via relay.build using the correct cross compiler and target object format as .o/.a.

I agree we should support binary output. My thought is this would depend on the TVM target in use–but we can talk about it. Either way, Model Library Format should work with source and/or binary.

I think its OK to bundle them in the model library as optional/helper sources but I think we should not link them in the compiled artifact (.o/.a). Yes, we would need a tvmc compile flag to specify the output type. I think we should discuss the contents of Model Library in that RFC

Let’s discuss this further in the RFC–ideally I’d like to leave anything specific to integration out, but perhaps there are some generic glue code we could include. In this particular situation–would it not make sense to treat bundle.{c,cc} as an implementation of the Project API?

@leandron wrote:

Do you see any other fundamental problems we are trying to solve with all being discussed in this thread, that is not captured here?

I think that is a fair summary of the different aspects here, but I’d like to point out one particular implication of P1:

  • by “improving tvmc compile to generate a more comprehensive set of outputs in the Model Library Format,” we are also proposing that the current µTVM compilation flow be split into two pieces: 1) generating Model Library Format and 2) building a project which can be compiled. This solves a fundamental problem of how to integrate tvmc with a variety of projects without adding all of their dependencies to TVM’s dependency list.

Finally, I’d say that depending on how we choose to implement Model Library Format, we should consider whether it should just become the output format from tvmc–but let’s discuss that on the RFC.

I agree we should support binary output. My thought is this would depend on the TVM target in use–but we can talk about it

Yeah I think so too. Like, I won’t be good to always generated a .so and a .o (for example) for the static runtime case, which afaics will only be “interested” in the .o to wrap it using the “adapters” (bundle.{c,cc}) to create a .so (dynamic case) or use link all the objects in a static executable (static case). In both cases I understand tvmc compile should not generate a .so.

Other use-cases, like for applications dlopen’ing the model library (.so) generated by the tvmc compile, it would be good to have .so as an output for command compile.

–would it not make sense to treat bundle.{c,cc} as an implementation of the Project API?

Yep, I’m wondering it too. It makes sense to have “adapters” like that implemented like a kind of project, so use the mechanism of the Project API.

Yes, I think that it would be a good implication, to have this unified format being generated by tvmc compile, that allows the output to be useful in hosted systems (for regular TVM), and RTOS-agnostic (in the case of uTVM), that makes it useful, as you pointed out, in a variety of project. I’m fully onboard with that.