Load dynamic module failed on Jetson nano with tvm.0.7.dev

Troubleshooting
#1

Hi Experts,

We compiled the model, then generated the json, params, and dynamic module so file.

but when loading the dynamic module so file, tvm failed:

#./TvmTestCases -url backham.jpg [11:50:12] /home/nano/Downloads/incubator-tvm/src/runtime/dso_library.cc:84: Check failed: lib_handle_ != nullptr: Failed to load dynamic shared library mnet12_face.so mnet12_face.so: cannot open shared object file: No such file or directory Stack trace: [bt] (0) /usr/local/lib/libtvm_runtime.so(+0x48df8) [0x7fb111cdf8] [bt] (1) /usr/local/lib/libtvm_runtime.so(tvm::runtime::Module::LoadFromFile(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)+0x3ac) [0x7fb1131a74]

First we thought i would be the reason: some dependencies can not be found.

So we use the ldd to check the dynamic module so file, but it shows:

#ldd mnet12_face.so

    statically linked

It is a statical module file!! And we can not load it.

Any sugguestions, Thanks in advance !!!

#2

Please check the format of mnet12_face.so. If this format is not correct arm format, you can not load it.

#3

Hi FrozenGene,

Thanks for your advice.

I check all compile and link tool are aarch64-linux-gnu, so i think it should be arm format.

Is there any details to check?

#4

We can find the dynamic modle format information:

#readelf -a mnet12_face.so | grep ‘Class|File|Machine’

Class: ELF64

Machine: AArch64

but can not read the dynamic information(usually there should be dependencies module information):

#readelf -d mnet12_face.so

Dynamic section at offset 0x89e10 contains 20 entries: Tag Type Name/Value 0x000000000000000c (INIT) 0x1560 0x000000000000000d (FINI) 0x25e70 0x0000000000000019 (INIT_ARRAY) 0x99e00 0x000000000000001b (INIT_ARRAYSZ) 8 (bytes) 0x000000000000001a (FINI_ARRAY) 0x99e08 0x000000000000001c (FINI_ARRAYSZ) 8 (bytes) 0x000000006ffffef5 (GNU_HASH) 0x1f0 0x0000000000000005 (STRTAB) 0xc10 0x0000000000000006 (SYMTAB) 0x478 0x000000000000000a (STRSZ) 2023 (bytes) 0x000000000000000b (SYMENT) 24 (bytes) 0x0000000000000003 (PLTGOT) 0x99fe8 0x0000000000000002 (PLTRELSZ) 48 (bytes) 0x0000000000000014 (PLTREL) RELA 0x0000000000000017 (JMPREL) 0x1530 0x0000000000000007 (RELA) 0x13f8 0x0000000000000008 (RELASZ) 312 (bytes) 0x0000000000000009 (RELAENT) 24 (bytes) 0x000000006ffffff9 (RELACOUNT) 3 0x0000000000000000 (NULL) 0x0

#5

First thing you could do is to save this shared library into one directory rather than tmp. For example, tvm_deploy directory. Then you could check the shared library in this tvm_deploy directory. I suggest this way is we could make sure our build target and compile tool work fine. Many times I am asked this question is he sets wrong target or wrong compile tool.

Second thing we could do is to check the directory we upload. I met this error another situation is I can not upload the shared library into the target directory.

#6

image
image986×342 15.1 KB

this is my compilation scripts.

As you mentioned, maybe the target is wrong.

It should be as following ?, i will try this:

target = tvm.target.cuda(“unknown”, “-target=aarch64-linux-gnu -libs=cudnn,cublas”)

#7

target_host is not correct. target_host should be llvm -target=aarch64-linux-gnu. target is using cudnn, which is you expect?

#8

Jetson is an nvidia gpu arm board.

so what i expected would be:

target = tvm.target.cuda(“unknown”, “-libs=cudnn,cublas”)

target_host = “llvm -target=aarch64-linux-gnu”

Hope this would work.

#9

Hi FrozenGene,

Thanks for your advice.

But nothing change.

ldd output the same text:statically linked and tvm can not load the module so.

I am really confused now.

#10

Firstly, let us remove -libs=cudnn,cublas and just use TVM cuda target and verify.

#11

Hi FrozenGene,

Thanks for your patience and concern.

It still not work.

And i think is there any option i missed?

#12

Could you verify CPU work or not? Then consider GPU.

#13

Yes, CPU compiled modle can read the dynamic information; but still failed to load(i am checking).

At least ldd can read the dynamic information.

#14

module compiled for CPU has ldd output information, but still can not be loaded(may be the path lead to failure).

nano@nano-videocap:~/Downloads/Inference/TestCases/TvmTestCases/bin$ ldd mnet12_face.so

    linux-vdso.so.1 (0x0000007f93970000)

    libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f937f7000)

    libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f9369e000)

    /lib/ld-linux-aarch64.so.1 (0x0000007f93945000)

nano@nano-videocap:~/Downloads/Inference/TestCases/TvmTestCases/bin$ ./TvmTestCases -url backham.jpg

[16:31:08] /home/nano/Downloads/incubator-tvm/src/runtime/dso_library.cc:84: Check failed: lib_handle_ != nullptr: Failed to load dynamic shared library mnet12_face.so mnet12_face.so: cannot open shared object file: No such file or directory

Stack trace:

[bt] (0) /usr/local/lib/libtvm_runtime.so(+0x48df8) [0x7f776fedf8]

bt/usr/local/lib/libtvm_runtime.so(tvm::runtime::Module::LoadFromFile(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&)+0x3ac) [0x7f77713a74]

[bt] (2) /home/nano/Downloads/Inference/TestCases/TvmTestCases/…/…/RetinaFace/Tvm/TvmFaceDetector/lib/libtvm_facedetector.so.1(inference::tvm::TvmInfer::ReadNetwork()+0xa8) [0x7f7a4ba0a8]

[bt] (3) /home/nano/Downloads/Inference/TestCases/TvmTestCases/…/…/RetinaFace/Tvm/TvmFaceDetector/lib/libtvm_facedetector.so.1(inference::tvm::TvmInfer::Create()+0x40) [0x7f7a4b9b20]

[bt] (4) /home/nano/Downloads/Inference/TestCases/TvmTestCases/…/…/RetinaFace/Tvm/TvmFaceDetector/lib/libtvm_facedetector.so.1(Tvm_Create(TvmFaceDetector**, sdk::image::ModelConfiguration const&, sdk::DeviceConfiguration const&, sdk::face::FaceParameter const&)+0x824) [0x7f7a4b15cc]

[bt] (5) ./TvmTestCases(main+0xbc) [0x557801ce64]

[bt] (6) /lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0xe0) [0x7f7a18a6e0]

[bt] (7) ./TvmTestCases(+0x85f8) [0x557801d5f8]

#15

Hi FrozenGene,

Thanks for your concern and all your advices so much.

This issue has been fixed(i write one small programe to test(dlopen) the usage of library), it is not the issue of TVM, but the limitation of nano system(arm ubuntu 18.04).

Thanks again for your support!!

1 Like
#16

Hello sorry to be bothering you but how did you end up solving this and being able to load this library in nvidia xavier?