How does system library work?

[kira-lin]

Hi all, I am trying to compile some TVM modules and export them for deployment in C++. I have tried the shared library approach and it works, but I don’t want to load the library at runtime. Therefore, I am checking the system library approach, but I found that it is unable to find my functions, seems like it’s empty. How does this system library work? I save it to a .o, and then pack it into my .so, with an all-in-one tvm runtime. Can I use export_library instead of save if I want to pack it into a .so?

[jinchenglee]

It depends on your runtime approach. It seems you try to get rid of the DLL part. One can try use static linking.

If you are trying the CRT runtime in tvm repository, you’ll have to do some homework to manually register your functions into the function registry before your program can find those functions.

[areusch]

hi @kira-lin, please take a look at apps/bundle_deploy which has an example of this.

Andrew

[rajivr]

Hi all,

Beginner question. Apologies if answered before. In the howto_deploy example, is there an example of prepare_graph_lib that uses system-lib? Our target does not support dynamic libs. The prepare_test_libs/DeploySingleOp works for me fine with system-libs. How do I do the same with prepare_graph_lib/DeployGraphRuntime

Thanks Rajiv

[areusch]

hi @rajivr,

You should be able to add -system-lib to the target string and get the tvm::runtime::Module with TVMFuncGetGlobal("runtime.SystemLib")().

Andrew

[rajivr]

Hi,

Thanks for your reply. I do get aborts with ‘bad_function_call’ when running. Pretty sure what I am doing must be wrong. So in prepare_test_libs.py, I see a call to export_library that generates the shared library. It does seem to generate two object files - lib0.o and devc.o and bundles them to a shared library. So with -system-lib, do I just save these individual object files and compile them with cpp_deploy.cpp and libtvm_runtime_pack.o?

And in cpp_deploy.cpp, I do see the below calls in DeployGraphRuntime:

tvm::runtime::Module gmod = mod_factory.GetFunction("default")(ctx); tvm::runtime::PackedFunc set_input = gmod.GetFunction("set_input"); tvm::runtime::PackedFunc get_output = gmod.GetFunction("get_output"); tvm::runtime::PackedFunc run = gmod.GetFunction("run");

How do I replace these with calls to TVMFuncGetGlobal?

[areusch]

@rajivr,

could you send a capture of your terminal output so I can see context around bad_function_call?

So with -system-lib, do I just save these individual object files and compile them with cpp_deploy.cpp and libtvm_runtime_pack.o

Yeah that’s right.

How do I replace these with calls to TVMFuncGetGlobal?

With -system-lib, you obtain a reference to gmod as follows:

tvm::runtime::Module gmod = TVMFuncGetGlobal("runtime.SystemLib")();

All following calls should be the same.

-Andrew

[rajivr]

@areusch

tvm::runtime::Module gmod = TVMFuncGetGlobal("runtime.SystemLib")();

When I do that, I get a compile time error:

cpp_deploy.cc: In function ‘void DeployGraphRuntime()’: cpp_deploy.cc:107:67: error: too few arguments to function ‘int TVMFuncGetGlobal(const char*, void**)’ 107 | tvm::runtime::Module gmod = TVMFuncGetGlobal("runtime.SystemLib")(); | ^ In file included from /scratch/rajivr/wsp/tvm_orig/tvm/include/tvm/runtime/module.h:30, from cpp_deploy.cc:25: /scratch/rajivr/wsp/tvm_orig/tvm/include/tvm/runtime/c_runtime_api.h:359:13: note: declared here 359 | TVM_DLL int TVMFuncGetGlobal(const char* name, TVMFunctionHandle* out);

Looks like it needs a handle to a PackedFunc?

[areusch]

@rajivr my apologies–my code wasn’t quite correct. try this:

tvm::runtime::Module gmod = tvm::runtime::Registry::Get("runtime.SystemLib")();

You can see a full example in apps/bundle_deploy/bundle.cc.

[rajivr]

It works fine now. Thanks for the quick replies.

[jinfagang]

How to generate a static lib?

[areusch]

@jinfagang could you clarify what you mean, since static library is different from (but related to) -system-lib? which pieces would you like in the static lib?

[jinfagang]

@areusch thanks god, there still have someone that can reply my question.

To be clear, I can using:

....
fadd_dylib = tvm.build(sched, [ph_a, ph_b, ph_c], tgt, name="vector_add")
    lib_path = 'libtvm_add.so'
    fadd_dylib.export_library(lib_path)

something like to export dynamic lib.

Now, I want export a static lib, which generates a .a file as you know.

I searched a little bit, seems there is something like this:

fadd_dylib.save('libadd.a')

But got this error

/tvm/src/target/llvm/llvm_module.cc", line 167
TVMError: Do not know how to save file libadd.a with format=''

tvm is so poor documentation (or I can not found maybe exist).

My final question is: Gen the static lib, and I can call it with this:

tvm::runtime::Module mod_tvmlib =
      (*tvm::runtime::Registry::Get("runtime.SystemLib"))();

in C++.

Any step by step tutorial for me?

[areusch]

Thanks for clarifying @jinfagang and apologies for the lack of documentation. Indeed, save and export_library can be a bit confusing. To clarify, TVM’s build function produces a tree of runtime::Module. Then:

• save exports a single one of those modules, and the supported format depends on the type of bitcode in that Module (e.g. for CSourceModules, .c format is supported; for LLVM, .o, .asm, etc format is supported). Typically a Module generates one .c or .o file.
• export_library exports all the Module in the tree and builds a library out of those artifacts.

I’d suggest a couple routes:

1. use mod.export_library("foo.tar"), which creates a tar archive containing those files. you can then compile those yourself into a static lib.
2. you might try mod.export_library("foo.a", fcompile=cc.create_executable), though I’m not sure if this will work.
3. For microTVM use cases when you are targeting c or llvm, you may try tvm.micro.export_model_library_format(mod), which creates an archive suitable for consumption by downstream compilation toolchains.

[jinfagang]

@areusch thanks for your reply.

I found a way to generate static lib now. I firstly same all lib to .o and then using ar compile them into static lib.

Also, I have changed the target to llvm --system-lib in te.build func.

Now, the question is that:

I can not retrive the func back in c++, here is how I get it:

tvm::runtime::Module mod_tvmlib =
      (*tvm::runtime::Registry::Get("runtime.SystemLib"))();
  printf("Done\n");
mod_tvmlib.GetFunction("vector_add");

Also, I added the stubs already:

TVM_FUNC_STATIC_STUB(vector_add)

Now the question becomes:

1. How to load .a back inside python? I want tested it on python first;
2. why my C++ load way can not found the func?

it would be very appreciated if there is any help, I really want tvm can have a compelete and easy to understand example for this.

[areusch]

hi @jinfagang, take a look at apps/bundle_deploy for an example of how to do this outside of Python. if you want to load your .a back into Python, you’ll need to modify the TVM build to include those .o files by changing the CMakeLists.txt (try appending them to TVM_RUNTIME_SRCS maybe or the equivalent OBJS list, which I can’t remember offhand right now). I think once you have loaded those files, you should be okay.

Regarding the error you’re seeing in c+±-assuming you have followed the method in bundle_deploy, you should be able to get the Module. it seeems like right now you may be getting a blank Module which is the default. TVM module initialization flow should be able to register symbols in the SystemLib via calling TVMBackendRegisterSystemLibSymbol. However, I’m wondering if linking with ar may have removed those init calls from the global ctors list, and whether you may need to instead call gcc to create the .a.

[jinfagang]

@areusch thanks for reply. I have solved all my questions.

[Civitasv]

I’m also trying to make apps/howto_deploy work, but get the same error:

Check failed: (f != nullptr) is false:

I also tried to use --system-lib option, but tvm tells me that it is deprecated now, how should I fix it now?

And I’m using unity version of TVM.