hi @cron,
glad you were able to solve the dladdr problems.
there are a couple of ways to run TVM operators in a bare-metal environment. a lot of the recent work has centered around creating a firmware binary that includes compiled TVM operators plus an RPC server which can be commanded to run them over e.g. a UART. this is slightly different from what you’re asking, but let me give you some pointers to that code since the generated operators are compatible (actually, exactly the same at present) between this approach and a fully standalone firmware binary:
-
first i’d point you to the recent test_zephyr.py as a starting point. I know you want to run without Zephyr–the generated code is agnostic to the presence or absence of Zephyr. you will need to provide some replacement platform hooks, but i’ve tried this and it’s not intended to be difficult.
-
another resource i’ll point you to is my recently uploaded virtual machine branch (soon, PR), which makes a couple changes to the transport layer to make things work with real hardware using a UART connection. it also checks in a Vagrant base box + another Vagrantfile, to be used together to bring up a local Virtual Machine with Zephyr + cross compilers for ARM, RISC-V, etc. I’ve tested this same script (test_zephyr.py) against both real hardware and QEMU on this branch.
now as to the question of creating a standalone binary: the main difference is that the TVM Graph Runtime needs to execute on the device. apps/bundle_deploy demonstrates a way to run Graph Runtime in a separate application from TVM; however, it relies on some linux filesystem calls to load data. you could run standalone on bare-metal by porting apps/bundle_deploy to your SoC. here’s roughly what I think you need to do to create a standalone µTVM firmware image that runs your model:
- as you mentioned, adjust the target string to
'c --system-lib -mcpu=cortex-mN' (you can follow the examples here). (you can also use llvm in place of c as bundle_deploy does, but this may be less readable).
- create a version of
apps/bundle_deploy/demo_static.c (note: only demo_static is worth trying to port, no other bundle_deploy targets are portable to bare metal) that loads input_storage using whichever way your platform provides. i’d imagine for a simple bare-metal ARM example, you could use just a flash array in the same way build_graph_c_json is handled. Presumably you could also read this from a sensor.
- create the binary firmware image: take the generated
.c (if using c target) or .o (if using llvm target) file created by apps/bundle_deploy/build_model.py, link it against your revised demo_static.c and the required runtime libraries in build/standalone_crt, specifically libcommon and libgraph_runtime.
I should also mention that test_zephyr.py is doing something slightly different from apps/bundle_deploy, but I’m giving instructions for something similar to bundle_deploy since that’s what you’re trying to do. specifically, test_zephyr.py is building a firmware image containing a TVM RPC server, then sending it commands to execute operators over a UART. the firmware created with test_zephyr.py cannot run a model without the PC connected.
here are some answers to your questions:
I know I have to change here to relay.build('llvm -mtriple=arm-none-eabi') so that the right llvm target machine is used. Anything else I need to set here?
answered above
Could you maybe elaborate on what --system-lib does?
see docs here
I also know that the Makefile requires some changes since it does not seem to account for 'arm-none-eabi’gcc prefix. Weirdly enough the Makefile for the crt libraries does accept this prefix. Am I missing something here?
yep, so bundle_deploy is not targeted to a cross-compiled nor bare-metal system. it’s meant to be compiled on the host it runs on, and expects linux for basic file operations. the actual model execution part in bundle_deploy's demo_static is portable. this part is made up of CRT libraries and the operator code generated by build_model.py. The Makefile for the CRT libraries is thus a little more flexible and can be retargeted with PREFIX=. Please make sure you’re using build/standalone_crt/Makefile when trying to use that build system–even though it’s the same file, you can’t use the Makefile placed at src/runtime/crt/Makefile because it expects certain paths to be placed in the correct spot (see cmake/modules/StandaloneCrt.cmake).
Could you help me with defining the flags required for compiling in this case? (I see that some flags are defined in the Makefile, but I cant imagine that those are enough for arm embedded bare metal target)
I can’t give too many specifics as I don’t know what SoC you’re compiling for. I’d say SoC-specific bringup will be hit-or-miss here, because it’s fairly specialize to your project–this is why we’ve opted to link against RTOS such as Zephyr in the hopes of producing something more broadly useful. Hopefully you can work off of some example code there to start with.
That said, feel free to ask if you have compilation or runtime problems and I or others will try and help as we can. I’m also particularly interested in talking about any improvements to the overall compilation flow or the CRT libraries that would make µTVM easier to use in everyone’s projects, since there will always be constraints from other projects that developers don’t know about.
Do you know of any reason why any of the modules required for the bundle or libraries are not bare metal compatible? (I tried to set up a compilation pipeline but I was getting newlib errors and it felt it had something to do with exception handling but I am not sure. I did however always try to compile with backtracing on).
No, the generated modules and CRT is bare-metal compatible. apps/bundle_deploy, however, defines 3 possible execution flows:
-
demo_static using CRT
-
demo_dynamic using CRT
-
demo_dynamic using C++ runtime (I.e. parts of libtvm_runtime.so)
if you try to link the code from the latter two, you’ll run into exception problems and problems with dlopen.
hope this helps–feel free to ask more questions and i’ll try and help as I can.
Andrew
