One of the challenges with using TVM as an ahead of time compiler is the lack of a simple interface to take a model and put it through the stack. The current solution of writing a python script has a few problems:
It leads to a lot of code duplication as everyone writes their own bit of python to accomplish the same task in a subtly different way.
It encourages writing special-cased solutions.
To even begin writing these python scripts requires a reasonably comprehensive knowledge of the API.
The last point in particular makes TVM fairly inaccessible to anyone who is not interested in becoming a TVM developer. We have produced some python scripts to expose the core functionality of the TVM stack via a command line interface.
tvm compile
Compile takes a model in some format (eg. a .tflite file) and compiles it down to a tvm.Module which is saved to disk as a tar file (containing the graph/lib/params). Here we can set compiler flags and dump intermediates.
tvm run
Run takes a compiled tvm.Module (as a tar file) and runs it over RPC. Input tensors can be filled with zeros/ones/random or alternatively a .npz file can be passed to initialise the inputs. The outputs can then also be saved as a .npz file. This generic approach lets us run many different types of network instead of just image classification. Additionally, ârunâ can time the execution and produce a profiling dump via the debug runtime.
tvm tune
Tune performs auto-tuning for a given model. This is the most speculative command as there exist a few special cases in tuning that are still best suited to a python script. However, the intention is to provide a âgood enoughâ route for someone who doesnât want to develop an intimate understanding of AutoTVM.
Usage
A typical workflow would likely consist of the following:
Following this RFC, if we are going to have a Python CLI with the interface as @liangfu illustrated, should also move all current CLI tools to the top level (e.g., python3 -m tvm rpc_server <args>)? It seems more general and reasonable in my opinion.
Given that different cli tools have their own set of arguments, i think it is still better to distinguish them, in terms of cost of typing, both are quite similar.
Thatâs a fair concern. FYI, a common way to distinguish commands with completely different functionalities is leveraging the hierarchical command groups such as git. For example:
python3 -m tvm rpc server <RPC server specific args>
python3 -m tvm tune <AutotVM specific args>
We can use argparse with add_subparsers to achieve this. Based on that, we can also provide a decorator for modules to register their CLIs to a certain command group. Something like
Our implementation uses subparsers with âtvmâ being the principle command (maybe we should come up with a better name). I agree with using a hierarchical approach and that we could put additional scripts behind a unified CLI with their own appropriate subcommand. Ideally weâd have this CLI be installable with pip or similar.