[µTVM] Capturing dependent libraries of code-generated TIR (initially for use in Model Library Format)

[areusch]

A bit of a rough draft which I started writing before last week’s µTVM discussion on including TVM C runtime inside Model Library Format tarballs and finished up this weekend. Comments welcomed!

cc @jroesch @tqchen @mousius @manupa-arm @ashutosh-arm @ramana-arm @comaniac @kparzysz @csullivan @mbs-octoml @tkonolige

Summary

This RFC proposes we add an external_dependencies key to Model Library Format metadata.json. Adding this key affords both users and downstream Project API servers a convenient way to know that the model must be linked against some third-party library in compilation. As we begin introducing support for third-party libraries such as CMSIS-NN in microTVM, having such a facility will become more useful.

Motivation

microTVM’s build flow departs significantly from the typical TVM build flow because it delegates all aspects of model compilation to the user. The microTVM build flow is roughly:

1. TVM exports the model of interest in Model Library Format, a TVM export format intended to include all of the relevant pieces needed to integrate a model with an e.g. firmware project.
2. The user integrates the pieces from the Model Library Format archive with their firmware project (either manually, or via Project API or some other automation).

This has currently been shown to work with models that target the CPU only as long as TVM selects a schedule that doesn’t depend on external libraries such as CMSIS-NN. In [RFC]Use CMSIS-NN with TVM, ARM is proposing to add CMSIS-NN schedules for many common operators. Some additional mechanism is needed in Model Library Format so TVM can indicate to downstream Project API and users that external libraries such as CMSIS-NN are required to compile the generated model code.

Guide-level explanation

A rough sketch of the design is as follows:

1. TVM can generate code dependent on third-party libraries in at least two cases:

   1. When a TVM or BYOC schedule introduces a tir.call_extern node which refers to a function implemented in an external library.
   2. When the code-loading process for a BYOC backend requires the use of an external library. For example, the CUDA codegen merely generates CUDA source. In the C++ runtime, CUDAModule is responsible for passing the source onward to the CUDA library for compilation. Though a parallel process has not yet been specified for the C runtime, we should consider this case when designing this RFC.

2. A path needs to be defined by which the BYOC or schedule can communicate the external dependency to the compiler. At present, the BYOC flow is a PackedFunc which accepts a Relay subgraph and returns a runtime::Module. However, given ongoing work in Additional Target Hooks RFC, it will soon be possible for BYOC to return TIR as well. Therefore, here again we have two cases to consider:

   1. When the schedule for an operator (e.g. from TVM-internal scheduling or from the new relay_to_tir pass) includes TIR that depends on an external library.

      • In this case, the schedule or BYOC should generate TIR containing an attribute kExternalDependency containing Array<ExternalDependency>.

   2. When the runtime::Module created in TIR codegen (e.g. due to use of a non-DSO-exportable backend with third-party library dependencies; or due to BYOC) requires a third-party library.

      • In this case, the runtime::Module should include a function get_external_dependencies which returns Array<ExternalDependency>. This function is included in the spirit of other such metadata-passing functions such as get_const_vars, which should likely be split into a separate metadata dict. Splitting metadata functions apart from operator implementations in runtime::Module is out of scope of this RFC.

3. The compiler flow then assembles a list of ExternalDependency structs:

   class ExternalDependencyNode : public ObjectNode {
       /*! \brief Short unique name for this external dependency.
        * All ExternalDependencyNode generated for a given Relay module with the same short_name
        * must match in all other attributes. The compiler validates this at the end of the flow.
        */
       std::string short_name;
   
       /*! \brief URL to the dependency. */
       std::string url;
   
       /*! \brief Type of URL. One of "path" "url" "git". */
       std::string url_type;
   
       /*! \brief Version specifier. Required for "git" url_type.
        * When url_type is "git," the tag or branch to checkout.
        */
       std::string version_spec;
   };
   

   As mentioned in the commentary, if two ExternalDependencyNode with the same short_name are generated within the same tvm.relay.build call, the other attributes must match. The compiler verifies this and removes exact duplicates.

4. The compiler returns external_dependencies as part of its output. Model Library Format surfaces this as a new key external_dependencies in metadata.json:

   {
       // ...
       "external_dependencies": [
           {
               "short_name": "cmsis-nn",
               "url": "http://github.com/...",
               "url_type": "git",
               "version_spec": "5.8.0"
           }
       ]
   }
   

Reference-level explanation

This section is brief as this pre-RFC seeks feedback on the Guide-level part. The main pieces needed:

1. A pass which finds all kExternalDependency attributes and assembles a list of them.
2. A collection function which invokes the pass and any get_external_dependencies functions, validates and uniquifies them, and creates the compiler output.
3. Model Library Format changes.

Drawbacks

This increases the maintenance burden on BYOC and internal non-DSO-exportable flows by requiring us to provide explicit versioning information about the libraries linked against.

Rationale and alternatives

• We could require that the dependency always be bundled in with the compiler output.
  • This seems worse as it increases compiler output size, and in many cases, the runtime already has taken care of this.
• We could specify less information e.g. only the name of the external dependency. This removes the burden of validating the uniquified list of external dependencies and removes one target-specific compilation failure mode.
  • While this is probably sufficient for people within the TVM community, downstream consumers of TVM output will probably be left with significant research needed to actually use the TVM-generated code.

Prior art

Don’t have any yet for this pre-RFC

Unresolved questions

1. Is this level of specificity the right level to use when linking to external dependencies?
2. How might we version or future-proof this metadata?
3. When generating artifacts for --runtime=c, the standalone_crt directory is in some sense an “external dependency” (e.g. it currently lives outside the Model Library Format tarball). Should we list it here as one?
4. (perhaps the subject of another RFC, but related yet here) Should we provide a pathway towards optionally including external dependencies inside Model Library Format tarballs? If we do, what is the C++ runtime equivalent, if any?
5. How should libtvm_runtime.so compile-time dependencies be represented here?

Future possibilities

Not yet considered.

[Mousius]

Hi @areusch,

This is an interesting topic, throwing in some of my personal thoughts here

One of the main concerns I have with this is introducing what becomes almost a package manager for a diverse range of dependencies. There have been a number of other attempts to standardise package management in the micro world, with mixed success, and getting this right across the broad range of libraries you’re suggesting (CMSIS to CUDA) presents a fairly large challenge - it’d be good to understand how much of a problem this is from others in the community.

@areusch

As we begin introducing support for third-party libraries such as CMSIS-NN in microTVM, having such a facility will become more useful.

I think it’s important to note that a lot of users will be completely fine integrating a MLF output without this metadata, if a user has an existing application they may well have an existing CMSIS bundled; it’d be good to avoid a scenario where TVM is presenting you with additional versions of libraries you’ve already vetted as working in your existing project that are compatible with the TVM generated code. In the case of the Project API, I can understand the motivation to automate this more, what are your thoughts on including this metadata or scripting to acquire packages in a project generator?

@areusch

• How might we version or future-proof this metadata?

I’d suggest we move to codifying the schema for the entire MLF metadata, in a format such as JSON Schema, so that we can validate it on exit to ensure the output is as we expect and that the schema itself is versioned.

@areusch

• When generating artifacts for --runtime=c, the standalone_crt directory is in some sense an “external dependency” (e.g. it currently lives outside the Model Library Format tarball). Should we list it here as one?

In my opinion, specificity of standalone_crt seems more constrained than the parameter --runtime. My suggestion would be that the runtime when specified would be included in a runtime folder and be treated as an artifact of the TVM generation rather than as an external dependency. This would suggest you could include a Rust runtime under the same runtime folder or whichever runtime you asked TVM for. Also, there should be an opt out such as --runtime-c-no-bundle to disable this behaviour if you have multiple MLFs.

[leandron]

I think that self-declaring dependencies in a metadata.json is a positive move in keeping track of the dependencies for MLF. I would like to comment on one specific point on the Unresolved questions section, which is related to the standalone_crt situation.

@areusch

When generating artifacts for --runtime=c, the standalone_crt directory is in some sense an “external dependency” (e.g. it currently lives outside the Model Library Format tarball). Should we list it here as one?

Currently, in order to integrate the outputs of an AoT project, there are includes mapping to the standalone_crt that the user can only obtain in one of two ways:

• Building TVM from scratch - not very user friendly
• Using a very hacky line to pull the path to an installed tlcpack wheel - not really an end-user solution

On this point, I think it is important to point out a fundamental difference between strictly external dependencies such as CMISIS-NN, that can be obtained elsewhere, widely documented, and standalone_crt, which is part of TVM, from the PoV of an end-user which might not be interested in the TVM internals, or using the means listed above.

It is important if we can provide a self-sufficient MLF tarball, including at least all the TVM bits needed.

With that workflow in mind, I think with a small tweak to your original metadata.json file, we can conciliate between having a neutral descriptor for self-declared dependencies and also give the user one artifact (the MLF tarball) the can be integrated in an embedded project.

Concretely, the idea is to have an entry that can point to dependencies included within the MLF, as mlf_path, that points to a directory within the tarball. Thinking on your previous example JSON, it would be something like the snippet below:

@areusch

The compiler returns external_dependencies as part of its output. Model Library Format surfaces this as a new key external_dependencies in metadata.json:

{
    // ...
    "external_dependencies": [
        {
            "short_name": "cmsis-nn",
            "url": "http://github.com/...",
            "type": "git",
            "version_spec": "5.8.0"
        },
        {
            "short_name": "standalone-crt",
            "path": "deps/standalone_crt",
            "type": "mlf_path",
            "version_spec": "<tvm version string>"
        }
    ]
}

Can we consider this option in this plan? Happy to do the work to setup this as an optional for MLF.

[areusch]

Thanks @mousius for the reply!

One of the main concerns I have with this is introducing what becomes almost a package manager for a diverse range of dependencies.

I think of this less as a package manager and more as documentation of what “should work” according to TVM. So if you’re using a CMSIS-NN schedule, you probably do want to know what version of CMSIS-NN is the one we expect to work. Then, it’s clear to you whether you’re going off the beaten track. I do think the package manager is significantly more involved with the particular RTOS and we shouldn’t aspire to that here.

I think it’s important to note that a lot of users will be completely fine integrating a MLF output without this metadata, … In the case of the Project API, I can understand the motivation to automate this more, what are your thoughts on including this metadata or scripting to acquire packages in a project generator?

I agree we hope that MLF are easy to manually integrate. In the case you want to run autotuning or provide a platform to an engineer with minimal firmware knowledge, an automated script becomes more useful. Could you elaborate on your question? I think people could include that if they wanted to e.g. call out to git clone <url> 3rdparty/<external_dependency_name> or find it locally on disk

I’d suggest we move to codifying the schema for the entire MLF metadata, in a format such as JSON Schema, so that we can validate it on exit to ensure the output is as we expect and that the schema itself is versioned.

I agree with that, but let us make that a separate RFC.

In my opinion, specificity of standalone_crt seems more constrained than the parameter --runtime . My suggestion would be that the runtime when specified would be included in a runtime folder and be treated as an artifact of the TVM generation rather than as an external dependency.

The reason I am sensitive to doing this is that it then becomes unclear what should reside in the MLF docs and where the boundaries of an MLF package are. Should I always presume to find standalone-crt in an MLF archive? What if we support the C++ runtime with MLF? MLF is firstly a model export format in my mind.

With that said, I do think there is a path to including external deps in an MLF if they are properly identified in metadata.json. I’ll follow up on that by replying to @leandron comment next.

[areusch]

@leandron thanks for posting this up!

I completely agree that standalone_crt is about as precious as the MLF tarball, since they should be version-matched. And, I agree that keeping them separate can be challenging to work with operationally. I think schedules could also potentially contribute other C code e.g. read_and_pad in the case of Cortex-M schedules which is also in the same boat here.

I am 100% okay with this plan. I think it would also be great to update Project API to remove the path to the MLF archive from generate_project should we adopt this approach. Does that seem reasonable to you? cc @gromero

[Mousius]

Hi @areusch,

Thanks for your reply, it helped clarify a lot, I’ve followed up on your questions below

@areusch

I agree we hope that MLF are easy to manually integrate. In the case you want to run autotuning or provide a platform to an engineer with minimal firmware knowledge, an automated script becomes more useful. Could you elaborate on your question? I think people could include that if they wanted to e.g. call out to git clone <url> 3rdparty/<external_dependency_name> or find it locally on disk

I think this answers my question in combination with the above answer, the project generator would take the metadata and download it automagically or the user would be responsible in finding it themselves?

@areusch

The reason I am sensitive to doing this is that it then becomes unclear what should reside in the MLF docs and where the boundaries of an MLF package are. Should I always presume to find standalone-crt in an MLF archive? What if we support the C++ runtime with MLF? MLF is firstly a model export format in my mind.

The C++ runtime is precisely why I motivated my example with a Rust runtime, should the user use --runtime=c then I would expect a C runtime in the archive under ./runtime in much the same way you’d expect C outputs in ./codegen with --target=c. Should you then decide to use --runtime=rust, it’d be a reasonable expectation to find a Rust runtime exported in the archive under ./runtime.

Hope that helps, it just makes sense to me as a user to expect the --runtime argument to correlate with the eventual archive and I think this works well in tandem with @leandron’s proposal to clarify which runtime you’ve received?

[gromero]

@areusch

I am 100% okay with this plan. I think it would also be great to update Project API to remove the path to the MLF archive from generate_project should we adopt this approach. Does that seem reasonable to you? cc @gromero

Hi @areusch @leandron !

So, I also agree that it’s good to keep track of the necessary standalone_crt C code for a compiled model in metadata.json under new proposed "external_dependencies" as Leandro suggested. Hence it’s nice to have the standalone_crt code pointed out explicitly inmetadata.json, with a version associated with it, as a kind of external dep etc.

@areusch However I think removing the MLF path from generate_project is at some extent a significant change in the API yes, and immediately raises the question about what should then populate the CRT in the new project dir? Would it be done by the Project API client user - not via the API anymore? Afaics Project API can completely resolve the standalone_crt dep and populate CRT in all the cases (it’s not a dep like CMSIS-NN, which is “truly external”), so why not keeping the path to the MLF archive in generate_project and make generate_project actually resolve that dep (search for standalone_crt in metadata.json) and populate the CRT accordingly in the new project dir based on the runtime packaged with the MLF?

[areusch]

@Mousius @gromero thanks for these replies. Some follow-ups below:

@Mousius

@areusch

I agree we hope that MLF are easy to manually integrate. In the case you want to run autotuning or provide a platform to an engineer with minimal firmware knowledge, an automated script becomes more useful. Could you elaborate on your question? I think people could include that if they wanted to e.g. call out to git clone <url> 3rdparty/<external_dependency_name> or find it locally on disk

I think this answers my question in combination with the above answer, the project generator would take the metadata and download it automagically or the user would be responsible in finding it themselves?

I think this could be implementation-defined. In general the idea is to provide enough information that the generator could acquire the source, but this might not always be possible (e.g. standalone_crt from an uncommitted TVM repo). The bare minimum requirement is that the external dependencies (and the versions used) should be documented and as unambiguous as is feasible from TVM’s perspective. The main goal here is to be able to give someone an MLF and they can verify they are integrating it with the correct dependencies–for example, a user may choose to ignore the version of CMSIS-NN listed, but they should be at least able to check this and determine they are doing this at their own risk.

@Mousius

@areusch

The reason I am sensitive to doing this is that it then becomes unclear what should reside in the MLF docs and where the boundaries of an MLF package are. Should I always presume to find standalone-crt in an MLF archive? What if we support the C++ runtime with MLF? MLF is firstly a model export format in my mind.

The C++ runtime is precisely why I motivated my example with a Rust runtime, should the user use --runtime=c then I would expect a C runtime in the archive under ./runtime in much the same way you’d expect C outputs in ./codegen with --target=c. Should you then decide to use --runtime=rust, it’d be a reasonable expectation to find a Rust runtime exported in the archive under ./runtime.

Hope that helps, it just makes sense to me as a user to expect the --runtime argument to correlate with the eventual archive and I think this works well in tandem with @leandron’s proposal to clarify which runtime you’ve received?

I’m open to placing the runtime at a different location than e.g. <model.tar>/external-deps. I chose standalone_crt for the name to match the build artifact and make it easier to correlate concepts. I do think it would be good to ensure that name makes it into the archive somewhere as it does identify a specific artifact from the TVM build. I’m open to labelling it differently. To make your proposal concrete, you’re suggesting to change the metadata.json entry like so?

        {
            "short_name": "tvm_runtime",
            "url": "./runtime/standalone_crt",
            "url_type": "mlf_path",
            "version_spec": "0.8.0"
        }

@gromero

@areusch However I think removing the MLF path from generate_project is at some extent a significant change in the API yes, and immediately raises the question about what should then populate the CRT in the new project dir?

Oh I’m sorry I made another typo. I meant to say “I think it would also be great to update Project API to remove the standalone_crt_dir argument.” I think this resolves your concern–let me know if not.

[gromero]

Oh I’m sorry I made another typo. I meant to say “I think it would also be great to update Project API to remove the standalone_crt_dir argument.” I think this resolves your concern–let me know if not.

ah! okay, it totally dissolves my concerns then fair enough, afaics it has no impact on the current Project API users, also considering the pytests

[Mousius]

@areusch

I’m open to placing the runtime at a different location than e.g. <model.tar>/external-deps. I chose standalone_crt for the name to match the build artifact and make it easier to correlate concepts. I do think it would be good to ensure that name makes it into the archive somewhere as it does identify a specific artifact from the TVM build. I’m open to labelling it differently. To make your proposal concrete, you’re suggesting to change the metadata.json entry like so?

        {
            "short_name": "tvm_runtime",
            "url": "./runtime/standalone_crt",
            "url_type": "mlf_path",
            "version_spec": "0.8.0"
        }

Close I’d go for something like:

        {
            "short_name": "tvm_standalone_crt",
            "url": "./runtime",
            "url_type": "mlf_path",
            "version_spec": "0.8.0"
        }

This has the nice effect of being in a standard path and having the metadata to tell you which dependency it is - I don’t believe there’s going to be a use case for loading multiple runtimes here?

[areusch]

right, that makes sense. doubtful, but the metadata.json schema would support that anyway if there was. theoretically the rust runtime may merely be drop-in compatible with the C runtime.

[ashutosh-arm]

Can we consider including paths to the sources and header files that a customer of MLF needs to link? It is possible that the advancing the versions, these paths could change in future for CMSIS-NN.

Also, at the other end of the spectrum is the use case where library sources are not available with the user of MLF. Codegen just generates APIs and only header file(s) is/are made available. Final linking with the library happens through .a / .o /. so.

[areusch]

We could, though this goes beyond the original stated purpose of documenting the versions needed. I do see your point though–depending on which e.g. CMSIS-NN functions are used, you may need to link more or less of the library. Would you like to propose a schema for this?

[ashutosh-arm]

Something like this?

  {
            "short_name": "cmsis-nn",
            "url": "http://github.com/...",
            "type": "git",
            "include_paths": ["CMSIS/NN/Include", "CMSIS/Core/Include"],
            "source_files": ["CMSIS/NN/Source/*/*.c", "Device/ARM/ARMCM55/Source/*.c"],
            "source_libs": [""],
            "version_spec": "5.8.0"
        }

Paths can be relative to the one provided via URL. Both the fields source_files and source_libs could be filled in / dropped by the MLF producer based on the use case at hand.

[areusch]

@ashutosh-arm do you envision sub-selecting the source files needed e.g. if only some operator impls are needed? we would need to develop a policy around how include_paths and source_files are fused from various operator implementations. I think that’s the hard part about adding this metadata to MLF. we may consider to do a first version to document the versions of tools needed and then add this later on as the mechanism for joining source_files becomes more clear.

[ashutosh-arm]

Yes, you are right. We don’t need this information on priority. It is a good to have feature though when the support for CMSIS-NN is completely in.