I was trying to do what @merrymercy suggested in this post
by generating .tar file. I could generate it, it has only lib.o file, no dev.cc is generated (referring to this url) am I missing something?.
I am trying to deploy the module in C++ application, I get the module by calling
tvm::runtime::Module mod_syslib = (*tvm::runtime::Registry::Get(“module._GetSystemLib”))();
But, how to load the parameters and the JSON file , any idea how to do this in C++?
Check if this tutorial helps here.
@srkreddy1238, thank you for sharing the url, but the example there is used to load .so files.
And when I replaced the line
tvm::runtime::Module mod_syslib = tvm::runtime::Module::LoadFromFile(“deploy.so”);
with
tvm::runtime::Module mod_syslib = (*tvm::runtime::Registry::Get(“module._GetSystemLib”))();
I got a runtime error
terminated called after throuwing an instance of ‘dmlc::Error’ what(): 00:54:19 ../../src/runtime/graph/graph_runtime.cc::541: Check failed pf != nullptr no such function in module: fuse_pad
note: no only fuse_pad is missing, all the operations in the JSON file are missing
Am I missing something when generating the library or loading the module?
dev.cc is used for gpu backend. For cpu only *.o is needed.
The link shared by @srkreddy1238 has a section about how to load *.o and json. You can double check.
@merrymercy, So I used the same code shared by @srkreddy1238, but used
tvm::runtime::Module mod_syslib = (*tvm::runtime::Registry::Get(“module._GetSystemLib”))();
instead, and I get the following error
terminated called after throuwing an instance of ‘dmlc::Error’ what(): 00:54:19 …/…/src/runtime/graph/graph_runtime.cc::541: Check failed pf != nullptr no such function in module: fuse_pad
I used “nm” and the fuse_pad is in the application, am I missing a build or function call step?
Check the code @srkreddy1238 linked. you shouldn’t remove
tvm::runtime::Module mod_syslib = tvm::runtime::Module::LoadFromFile(“deploy.so”);
After you load dso module mod_syslib by the line above, you can get actual runtime module by
tvm::runtime::Module mod = (*tvm::runtime::Registry::Get("tvm.graph_runtime.create"))(json_data, mod_syslib, device_type, device_id);
And LoadFromFile can only load shared libs, because it uses dlopen (unix) or LoadLibrary (win). If you are trying to load static lib dynamically at runtime, you are doing it wrong.
In this question, I am not extracting shared library .so file, I build with extracted .o file, I do this in python with this line
lib_path = os.path.join(project_root, 'unet_deploy_arm.tar')
lib.export_library(lib_path,fcompile=False)
graph_json_path = os.path.join(project_root, 'unet.json')
with open(graph_json_path, 'w') as fo:
fo.write(graph.json())
param_path = os.path.join(project_root, 'unet.params')
with open(param_path, 'wb') as fo:
fo.write(nnvm.compiler.save_param_dict(params))
Then in a C++ application I do this:
tvm::runtime::Module mod_syslib = (*tvm::runtime::Registry::Get("module._GetSystemLib"))();
// json graph
std::ifstream json_in("unet.json", std::ios::in);
std::string json_data((std::istreambuf_iterator<char>(json_in)), std::istreambuf_iterator<char>());
json_in.close();
// parameters in binary
std::ifstream params_in("unet.params", std::ios::binary);
std::string params_data((std::istreambuf_iterator<char>(params_in)), std::istreambuf_iterator<char>());
params_in.close();
// parameters need to be TVMByteArray type to indicate the binary data
TVMByteArray params_arr;
params_arr.data = params_data.c_str();
params_arr.size = params_data.length();
int dtype_code = kDLFloat;
int dtype_bits = 32;
int dtype_lanes = 1;
int device_type = kDLCPU;
int device_id = 0;
// get global function module for graph runtime
tvm::runtime::Module mod = (*tvm::runtime::Registry::Get("tvm.graph_runtime.create"))(json_data, mod_syslib, device_type, device_id);
std::cout << "Graph created"<<std::endl;
I cross-compile the source code above, and link it with the lib.o in the .tar file.
I get the error by calling tvm.graph_runtime.create, I suppose, loading the module was not done correctly, any idea how to debug this?
Do you see the symbols (fuse_xxxx …etc) in the compiled binary ?
You may try nm or objdump to check for the fuse_xxxx symbols and the definitions.
Yes I see all the operations in the binary using nm.
Also please check of the symbol “TVMBackendRegisterSystemLibSymbol” present ?
Is it possible to share the compiled binary to inspect ?
Also what is your environment ?
CPU / GPU ?
X86 / Cross-Compile ?
Build system type (make / bazel …etc)?
TVMBackendRegisterSystemLibSymbol exists in the binary
I will try to share the binary
CPU is ARM A57
Cross-compile Linaro GCC v5.2
make
I’m facing the same issue, but only on Ubuntu.
I’ve generated a compiled.o with target
llvm --system-lib,
linked it into the test app under apps/howto_deploy, and got a tvm module with *tvm::runtime::Registry::Get("module._GetSystemLib").
However I can only get the packedFuncs from the module on MacOS, where I have brew installed llvm 6.0.1 and clang. On Ubuntu (16.04, GCC 5.4.0, LLVM 6.0.1) I just got the same “empty pf” assert.
It seems that on MacOS, system_lib_module.cc:TVMBackendRegisterSystemLibSymbol() gets invoked multiple times to register the symbols in compiled.o, which is then used to populate tbl_. On Ubuntu the runtime doesn’t enter this function at all, and tbl_ remains empty.
I’m unable to find an entrypoint to TVMBackendRegisterSystemLibSymbol() that can explain this different behavior, Thoughts?
@kgomaa.
Did you solve this issue.?
facing the same issue while loading static lib.
@srkreddy1238 any help here?
@Vinayak618 for loading Param and JSON to a statically linked application you can refer to the example in repo.
Thanks @kgomaa.
I’ll try that.
Thanks @kgomaa.
It worked like a charm.
Thanks alot