How to use Arm compute library with cpp RPC?

darkrai · September 16, 2021, 8:31am

Dear All, The question likes How to use Arm compute library with android RPC?

I work on x86 and Android.When I close ACL, USE_ARM_COMPUTE_LIB_GRAPH_RUNTIME and USE_ARM_COMPUTE_LIB both OFF, USE_CPP_RPC ON.I can compile out libtvm_runtime.so and tvm_rpc.I push them on Android and run, then I can use tracker in x86 to remote interaction with android.

But, when I use ACL, I meet some questions.I use USE_ARM_COMPUTE_LIB_GRAPH_RUNTIME ON,and I choose arm_compute-v20.11-bin-android.I made libtvm_runtime.so and tvm_rpc,but when i push them on device and try to run tvm_rpc, Error message:

CANNOT LINK EXECUTABLE "/data/local/tmp/tvm_rpc": cannot locate symbol "_ZTVN11arm_compute18NEConvolutionLayerE" referenced by "/data/local/tmp/libtvm_runtime.so"...

I don’t knwo how to fix it.Please help me, Thank you!

fantasyRqg · September 22, 2021, 9:34am

Enable USE_ARM_COMPUTE_LIB in config.cmake

darkrai · September 16, 2021, 1:27pm

Thanks，But I fixed this problem, another problem come. when I run tvm_rpc on device the Error message is:

terminating with uncaught exception of type std::bad_cast: std::bad_cast

darkrai · September 16, 2021, 1:58pm

Because of https://github.com/apache/tvm/pull/7649, I fix the ACL version to 21.02. But the error message doesn’t disappear：

terminating with uncaught exception of type std::bad_cast: std::bad_cast

Thanks!

darkrai · September 22, 2021, 8:38am

It still not working.Has anyone used ACL and cpp rpc on andorid？

Thanks!

fantasyRqg · September 22, 2021, 9:43am

Your error mesage seems like a compile error message. I’m not sure.

Workflow, not verified

Download ACL lib from ACL Release according to your device arch.
Extract lib and set absolute path in your tvm config.came file USE_ARM_COMPUTE_LIB
Cross compile tvm_rpc target
Push acl libs and libtvm_runtime.so to your device.
export LD_LIBRARY_PATH=path_to_your_libs. This environment variable tells system where find your dynamic libs.
run tvm_rpc and test

darkrai · September 22, 2021, 2:16pm

Thanks for your reply! What I did is exactly what you said, but I did’nt work.

Now, I change some 3rdparty version and fixed std::bad_cast!but when I use rpc to run remote module, I meet new error message. Now I use

acl-v21.02, android-arm64-v8a-cl
llvm-6.0.0
android-ndk-r21e

new errer message:

File "/root/.local/lib/python3.8/site-packages/tvm-0.8.dev0-py3.8-linux-x86_64.egg/tvm/contrib/graph_runtime.py", line 206, in run
    self._run()
File "tvm/_ffi/_cython/./packed_func.pxi", line 322, in tvm._ffi._cy3.core.PackedFuncBase.__call__
File "tvm/_ffi/_cython/./packed_func.pxi", line 257, in tvm._ffi._cy3.core.FuncCall
File "tvm/_ffi/_cython/./packed_func.pxi", line 246, in tvm._ffi._cy3.core.FuncCall3
File "tvm/_ffi/_cython/./base.pxi", line 160, in tvm._ffi._cy3.core.CALL
tvm._ffi.base.TVMError: Traceback (most recent call last):
[bt] (5) /root/.local/lib/python3.8/site-packages/tvm-0.8.dev0-py3.8-linux-x86_64.egg/tvm/libtvm.so(TVMFuncCall+0x63) [0x7fa600f12213]
[bt] (4) /root/.local/lib/python3.8/site-packages/tvm-0.8.dev0-py3.8-linux-x86_64.egg/tvm/libtvm.so(+0x2bb5517) [0x7fa600f81517]
[bt] (3) /root/.local/lib/python3.8/site-packages/tvm-0.8.dev0-py3.8-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::RPCWrappedFunc::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const+0x365) [0x7fa600f87e15]
[bt] (2) /root/.local/lib/python3.8/site-packages/tvm-0.8.dev0-py3.8-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::RPCClientSession::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)> const&)+0x57) [0x7fa600f79e27]
[bt] (1) /root/.local/lib/python3.8/site-packages/tvm-0.8.dev0-py3.8-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::RPCEndpoint::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)>)+0x5d1) [0x7fa600f6e181]
[bt] (0) /root/.local/lib/python3.8/site-packages/tvm-0.8.dev0-py3.8-linux-x86_64.egg/tvm/libtvm.so(+0x2b9efa8) [0x7fa600f6afa8]
File "/data/clean/tvm/src/runtime/rpc/rpc_endpoint.cc", line 798
TVMError: 
---------------------------------------------------------------
An internal invariant was violated during the execution of TVM.
Please read TVM's error reporting guidelines.
More details can be found here: https://discuss.tvm.ai/t/error-reporting/7793.
---------------------------------------------------------------
Check failed: code == RPCCode::kReturn == false: code=1

Thanks for help!

fantasyRqg · September 23, 2021, 9:15am

Test passed

Prepare Host

On host, set USE_ARM_COMPUTE_LIB to ON, and set USE_ARM_COMPUTE_LIB_GRAPH_EXECUTOR to OFF. Build host tvm runtime

Prepare Android Device

Clone above repo.
Modify TVM_Android_rpc/app/src/main/cpp/CMakeLists.txt , set TVM_ROOT to your tvm absolute path.
Build and install app.
Setup your tvm rpc tracker , config ip and port in app. click Start
Use android_acl.py to reproduce result. Pay attention to modify ip and port.

Upload...
/var/folders/7w/k_970t_90wd9vtchpjdqhnmw0000gn/T/tmp7irbuxjl/net.so
[[[[0.7889827  0.8712346  0.7862101  ... 0.5547441  0.9647179
    0.5783195 ]
   [0.6834558  0.8510778  0.6998373  ... 0.75880164 0.93593764
    0.7739675 ]
   [0.7384401  0.86287856 0.76005685 ... 0.88722074 0.7189027
    0.8670269 ]
   ...
   [0.6210304  0.7024486  0.78508025 ... 0.70067745 0.2930827
    0.74599665]
   [0.7948077  0.7144487  0.9428007  ... 0.8444273  0.97675955
    0.5060871 ]
   [0.73674494 0.8966101  0.6047768  ... 0.5997092  0.659487
    0.976786  ]]

  [[0.6582405  0.8955991  0.7946559  ... 0.96409416 0.30086812
    0.90660506]
   [0.87152964 0.8748835  0.9879059  ... 0.92298627 0.8041229
    0.8012321 ]
   [0.9582267  0.8447449  0.9845872  ... 0.91150975 0.9477019
    0.77571946]
   ...
   [0.8753126  0.9122753  0.9475235  ... 0.52288675 0.91917366
    0.80569947]
   [0.7897056  0.59507066 0.95199686 ... 0.98582304 0.39907318
    0.97375757]
   [0.8006658  0.8627856  0.8587522  ... 0.84566516 0.9859083
    0.93576187]]

  [[0.69935656 0.6610488  0.77729744 ... 0.95808613 0.8871862
    0.70124507]
   [0.96194035 0.8466262  0.7816765  ... 0.8726061  0.8633321
    0.83798146]
   [0.9959121  0.592816   0.8411254  ... 0.99859583 0.89235914
    0.9211045 ]
   ...
   [0.46039706 0.94635993 0.8654158  ... 0.60947555 0.6821387
    0.65045184]
   [0.7594721  0.5819088  0.80962765 ... 0.9767588  0.9679796
    0.99542665]
   [0.9529131  0.87899905 0.67325413 ... 0.9388463  0.95910054
    0.9331764 ]]

  ...

  [[0.82952344 0.8912541  0.6244757  ... 0.7723744  0.6033241
    0.8566712 ]
   [0.99556583 0.5525235  0.92290735 ... 0.87749183 0.71205854
    0.93232554]
   [0.9762415  0.7490494  0.90305126 ... 0.7754237  0.2926893
    0.6070457 ]
   ...
   [0.9564342  0.6525259  0.9860386  ... 0.6268005  0.5185014
    0.78197604]
   [0.83606607 0.80195034 0.69601256 ... 0.9483992  0.8886304
    0.7920514 ]
   [0.82950675 0.70211756 0.94573724 ... 0.9678505  0.6554203
    0.6589762 ]]

  [[0.9869032  0.88991576 0.9275303  ... 0.9087553  0.9885103
    0.9962867 ]
   [0.656542   0.9628975  0.60307556 ... 0.86081076 0.54222804
    0.66637665]
   [0.5881173  0.9210938  0.8600825  ... 0.9994548  0.5762127
    0.8954903 ]
   ...
   [0.6432559  0.9351148  0.9602096  ... 0.9561389  0.7673705
    0.36411133]
   [0.9139243  0.9439198  0.9346934  ... 0.87135    0.16547936
    0.85842496]
   [0.78570634 0.86811525 0.6606961  ... 0.9817263  0.9290371
    0.8340611 ]]

  [[0.97417134 0.6254694  0.592295   ... 0.77769184 0.9343401
    0.9535813 ]
   [0.54411626 0.9675265  0.9873329  ... 0.7878718  0.6518608
    0.93273246]
   [0.92705035 0.8961658  0.5183149  ... 0.7751632  0.7658381
    0.8775598 ]
   ...
   [0.5157349  0.9345443  0.9499851  ... 0.9740264  0.2355156
    0.9584289 ]
   [0.8735287  0.9636269  0.7376651  ... 0.72966486 0.7129484
    0.8810908 ]
   [0.95522004 0.8620265  0.95172983 ... 0.68910736 0.48140037
    0.5211506 ]]]]

Process finished with exit code 0

darkrai · September 24, 2021, 12:45pm

Thank you very much!

But I have something wrong with android rpc…Have you tried cpp rpc ？https://github.com/apache/tvm/tree/main/apps/cpp_rpc

fantasyRqg · September 24, 2021, 1:08pm

You should look source code. I used source code of cpp_rpc. For ease of use, build this app.

github.com

fantasyRqg/TVM_Android_rpc/blob/c65849b66409cdd997d4f27c5a48f6620edd2aea/app/src/main/cpp/CMakeLists.txt#L10


cmake_minimum_required(VERSION 3.10.2)


set(CMAKE_CXX_STANDARD 14)


set(TVM_ROOT /Users/rqg/test/tvm/)
include_directories(${TVM_ROOT})
include_directories(${TVM_ROOT}/include)
include_directories(${TVM_ROOT}/3rdparty/dlpack/include)
include_directories(${TVM_ROOT}/3rdparty/dmlc-core/include)
include_directories(${TVM_ROOT}/apps/cpp_rpc)
include_directories(include)


link_directories(libs/${CMAKE_ANDROID_ARCH_ABI})


add_definitions(-DDMLC_USE_LOGGING_LIBRARY=<tvm/runtime/logging.h>)
add_definitions(-DTVM_USE_LIBBACKTRACE=0)
add_definitions(-DTVM_LOG_CUSTOMIZE=1)
add_definitions(-DTVM_GRAPH_EXECUTOR_ARM_COMPUTE_LIB)


add_library(tvm_glue

TVM_Android_rpc just an app version of cpp_rpc

darkrai · September 26, 2021, 1:00pm

Hello, I run your code successfully.Thanks!

But when I use pytorch and tvm, like this demo, run resnet18 demo,It still crash. Please help me to find the reason.I submit a pr, https://github.com/fantasyRqg/TVM_Android_rpc/pull/1

Thank you very much!

lhutton1 · September 28, 2021, 8:32am

Hi @darkrai,

ACL was only intended to support NHWC format graphs, as this is the layout ACL uses internally. PyTorch, on the other hand, uses NCHW as a default format. TFLite graphs graphs would be advised as this is what we targeted when developing the integration.

That being said, I wouldn’t expect ACL to cause a failure, the non-compatible operations should simply not be offloaded to the ACL backend. In your case I would expect this to include the whole graph.

It’s been a while since I looked at the ACL integration, perhaps me and/or @dmitriy-arm could take a look at this when we have some free time.