[Hexagon] running model on hexagon cdsp getting "undefined symbol" error

Troubleshooting
#1

Hi all, I’m trying to run a simple model on hexagon dsp following these step to build “launcher_android” and “libtvm_runtime.so”:

cmake -DCMAKE_TOOLCHAIN_FILE="/data00/android-ndk-r25c/build/cmake/android.toolchain.cmake"
-DANDROID_ABI=arm64-v8a
-DANDROID_PLATFORM=android-28
-DUSE_HEXAGON_SDK="/data00/5.3.0.0"
-DUSE_HEXAGON_ARCH=v68
-DCMAKE_CXX_STANDARD=17
-DTVM_LOG_CUSTOMIZE=1
/data00/tvm/apps/hexagon_launcher/cmake/android
&& make -j8

and for “liblauncher_rpc_skel.so” I use:

cmake -DCMAKE_C_COMPILER="/data00/ziyeyang/5.3.0.0/tools/HEXAGON_Tools/8.6.06/Tools/bin/hexagon-clang"
-DCMAKE_CXX_COMPILER="/data00/5.3.0.0/tools/HEXAGON_Tools/8.6.06/Tools/bin/hexagon-clang++"
-DUSE_HEXAGON_ARCH=v68
-DUSE_HEXAGON_SDK="/data00/5.3.0.0"
-DCMAKE_CXX_STANDARD=17
-DTVM_LOG_CUSTOMIZE=1
/data00/tvm/apps/hexagon_launcher/cmake/hexagon
&& make -j8

when I trying to run on device using: /launcher_android --in_config gan_input.json --out_config output.json

got the following error: error loading graph module: 4e

and in logcat:

01-01 08:03:31.970 8120 8123 V adsprpc : CDSP:fastrpc_uprocess.c:733:0x567:133: fastrpc_uthread_create done for tidQ 0x562, tidA 8123 (/frpc/f047ed20 1314914_qvm), stack 0x00E70060 (size 131072), prio 192 01-01 08:03:32.022 8120 8123 V adsprpc : CDSP:rtld.c:1405:0x564:133: RX VA 0xFF300000 outside ELF segment 01-01 08:03:32.022 8120 8123 V adsprpc : CDSP:rtld.c:1405:0x564:133: RX VA 0xFF300000 outside ELF segment 01-01 08:03:32.023 8120 8123 V adsprpc : CDSP:rtld.c:1405:0x564:133: RX VA 0xFF300000 outside ELF segment 01-01 08:03:32.024 8120 8123 V adsprpc : CDSP:rtld.c:1405:0x564:133: RX VA 0xFF300000 outside ELF segment 01-01 08:03:32.028 8120 8123 V adsprpc : CDSP:mod_table.c:690:0x564:133: open_mod_table_open_dynamic: Module launcher_rpc_skel_handle_invoke opened with handle 0xea1e40 01-01 08:03:32.037 8120 8123 V adsprpc : CDSP:/data00/ziyeyang/tvm/src/runtime/dso_library.cc:133:0x564:133: Model .so Start Address : 20020000 01-01 08:03:32.037 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “__TVMBackendAllocWorkspace” 01-01 08:03:32.037 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “__TVMBackendFreeWorkspace” 01-01 08:03:32.037 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “__TVMBackendParallelLaunch” 01-01 08:03:32.037 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “__TVMBackendParallelBarrier” 01-01 08:03:32.037 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “__tvm_dev_mblob” 01-01 08:03:32.039 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.039 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.039 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.039 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.039 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.039 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.039 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.040 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.040 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.040 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.040 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.040 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.040 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.040 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param” 01-01 08:03:32.040 8120 8123 V adsprpc : CDSP:rtld.c:1057:0x564:133: Undefined symbol “_lookup_linked_param”

Does anyone knows how to solve this? thanks in advance

#2

When you load a compiled model using dlopen (as the launcher_android does), there are usually some undefined symbol references in it, but they are generally harmless. This especially applies to the double-underscored symbols like __TVMBackendAllocWorkspace. I’m not sure about the _lookup_linked_param though.

How did you build the model? Did you use the AoT executor?

Edit: Could you post more output from logcat? Especially the output leading to the error 4e?

#3

We didn’t use the AoT actuator, would it make a difference? The steps to build the model refer to tvm/apps/hexagon_launcher at main · apache/tvm · GitHub. A very simple onnx model with fixed I/O shape. The full log is as follows:

01-01 09:30:05.970 22504 22504 I launcher_android: vendor/qcom/proprietary/adsprpc/src/rpcmem_android.c:159: rpcmem_init_internal: opened ION device fd 4, configured heap IDs: system (0x2000000), contig (0x10), secure (0x400), secure flags (0x80080000)
01-01 09:30:05.971 22504 22504 I launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:3087: fastrpc_apps_user_init done with default domain:3 and &fastrpc_trace:0x7b6bed90b4
01-01 09:30:05.971 22504 22504 I launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2039: remote_session_control DSP info request for domain 3, thread priority -1, stack size 131072
01-01 09:30:05.971 22504 22504 I launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_config.c:200: Reading configuration file: launcher_android.debugconfig
01-01 09:30:05.998 22504 22504 I launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:2843: Created user PD on domain 3 (attrs 0x8, debug_trace 0x0)
01-01 09:30:06.000 22504 22505 I launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:484: Successfully set remote user thread priority to 192 and stack size to 131072 for domain 3
01-01 09:30:06.000 22504 22505 I launcher_android: vendor/qcom/proprietary/adsprpc/src/listener_android.c:111: listener thread starting
01-01 09:30:06.001 22504 22504 I launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_perf.c:273: fastrpc_perf_init: enabled systrace 0x0 and RPC traces (kernel 0, dsp 0) with frequency 1000
01-01 09:30:06.002 22504 22506 I launcher_android: vendor/qcom/proprietary/adsprpc/src/log_config.c:345: file_watcher_thread starting for domain 3
01-01 09:30:06.002 22504 22505 E ion     : ioctl c0044901 failed with code -1: Inappropriate ioctl for device
01-01 09:30:06.004 22504 22505 I launcher_android: vendor/qcom/proprietary/adsprpc/src/mod_table.c:687: open_mod_table_open_from_static: reverse module apps_std opened with handle 0x6bedb7e0 (idx 0)
01-01 09:30:06.005 22504 22505 I launcher_android: vendor/qcom/proprietary/adsprpc/src/mod_table.c:687: open_mod_table_open_from_static: reverse module apps_mem opened with handle 0x6bedb8e0 (idx 1)
01-01 09:30:06.005 22504 22505 I launcher_android: vendor/qcom/proprietary/adsprpc/src/apps_std_imp.c:879: Successfully opened file /data/local/tmp/dsp_selfbuilt/liblauncher_rpc_skel.so
01-01 09:30:06.009 22504 22506 E launcher_android: vendor/qcom/proprietary/adsprpc/src/log_config.c:193:Found launcher_android.farf. adspmsgd enabled 
01-01 09:30:06.010 22504 22507 V adsprpc : CDSP:fastrpc_uprocess.c:733:0x1567:645: fastrpc_uthread_create done for tidQ 0x1562, tidA 22507 (/frpc/f04777b0 1454190_qvm), stack 0x00E70060 (size 131072), prio 192
01-01 09:30:06.021 22504 22505 I launcher_android: vendor/qcom/proprietary/adsprpc/src/apps_std_imp.c:879: Successfully opened file /data/local/tmp/dsp_selfbuilt/libc++.so.1
01-01 09:30:06.028  4180  5475 W reserveLengthFromCarPos: : 25
01-01 09:30:06.028  4180  5475 W dynamic arrow config is null: ERROR
01-01 09:30:06.028  4180  5510 W state_hd_sd: 0
01-01 09:30:06.030  2898  3236 D cdc_sensorservice: sensor_service_impl.cc:OnIMUDataEvent:86 "IMUData:IMUData{timeStamp: 5411379202, imuState: 1, caliState: 1, px: 0.000698, py: 0.005585, pz: -0.001047, ax: -0.269683, ay: -0.142196, az: 10.140077, car_speed: -1, car_turn_angle: 0.000000, car_turn_speed: 0.000000}"
01-01 09:30:06.031 22504 22505 I launcher_android: vendor/qcom/proprietary/adsprpc/src/apps_std_imp.c:879: Successfully opened file /data/local/tmp/dsp_selfbuilt/libc++abi.so.1
01-01 09:30:06.064 22504 22507 V adsprpc : CDSP:rtld.c:1405:0x1564:645: RX VA 0xFF300000 outside ELF segment
01-01 09:30:06.064 22504 22507 V adsprpc : CDSP:rtld.c:1405:0x1564:645: RX VA 0xFF300000 outside ELF segment
01-01 09:30:06.065 22504 22507 V adsprpc : CDSP:rtld.c:1405:0x1564:645: RX VA 0xFF300000 outside ELF segment
01-01 09:30:06.065 22504 22507 V adsprpc : CDSP:rtld.c:1405:0x1564:645: RX VA 0xFF300000 outside ELF segment
01-01 09:30:06.070 22504 22507 V adsprpc : CDSP:mod_table.c:690:0x1564:645: open_mod_table_open_dynamic: Module launcher_rpc_skel_handle_invoke opened with handle 0xea1e40
01-01 09:30:06.070 22504 22504 I launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:1401: remote_handle64_open: Successfully opened handle 0x868203d0 for file:///liblauncher_rpc_skel.so?launcher_rpc_skel_handle_invoke&_modver=1.0&_dom=cdsp on domain 3
01-01 09:30:06.071 22504 22505 I launcher_android: vendor/qcom/proprietary/adsprpc/src/apps_std_imp.c:859: Successfully opened file linear.so
01-01 09:30:06.077  4180  5475 W reserveLengthFromCarPos: : 25
01-01 09:30:06.077  4180  5475 W dynamic arrow config is null: ERROR
01-01 09:30:06.077  4180  5510 W state_hd_sd: 0
01-01 09:30:06.081 22504 22507 V adsprpc : CDSP:/data00/ziyeyang/tvm/src/runtime/dso_library.cc:133:0x1564:645: Model .so Start Address : 20004000
01-01 09:30:06.081 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "__TVMBackendAllocWorkspace"
01-01 09:30:06.081 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "__TVMBackendFreeWorkspace"
01-01 09:30:06.081 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "__TVMBackendParallelLaunch"
01-01 09:30:06.081 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "__TVMBackendParallelBarrier"
01-01 09:30:06.081 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "__tvm_dev_mblob"
01-01 09:30:06.083 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "_lookup_linked_param"
01-01 09:30:06.083 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "_lookup_linked_param"
01-01 09:30:06.083 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "_lookup_linked_param"
01-01 09:30:06.083 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "_lookup_linked_param"
01-01 09:30:06.083 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "_lookup_linked_param"
01-01 09:30:06.083 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "_lookup_linked_param"
01-01 09:30:06.080     0     0 E cyttsp6_i2c_adapter 3-0024: cyttsp6_wait_bl_heartbeat: tmo waiting bl heartbeat cd->mode=0
01-01 09:30:06.080     0     0 E cyttsp6_i2c_adapter 3-0024: cyttsp6_startup_: Error on h/w reset r=-62
01-01 09:30:06.083 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "_lookup_linked_param"
01-01 09:30:06.083 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "_lookup_linked_param"
01-01 09:30:06.083 22504 22507 V adsprpc : CDSP:rtld.c:1057:0x1564:645: Undefined symbol "_lookup_linked_param"
01-01 09:30:06.088 22504 22507 E launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:1222: Error 0x4e: remote_handle64_invoke failed for handle 0x86820010, method 5 on domain 3 (sc 0x5000100) (errno Success)
01-01 09:30:06.089 22504 22505 E launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:1200: Error 0x4e: remote_handle_invoke failed for handle 0x3, method 4 on domain 3 (sc 0x4020200) (errno Operation not permitted)
01-01 09:30:06.089 22504 22507 E launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:1222: Error 0x4e: remote_handle64_invoke failed for handle 0x86820010, method 5 on domain 3 (sc 0x5000100) (errno Operation not permitted)
01-01 09:30:06.089 22504 22504 E launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:1222: Error 0x4e: remote_handle64_invoke failed for handle 0x868203d0, method 2 on domain 3 (sc 0x2030000) (errno Operation not permitted)
01-01 09:30:06.089 22504 22505 E launcher_android: vendor/qcom/proprietary/adsprpc/src/fastrpc_apps_user.c:1200: Error 0x4e: remote_handle_invoke failed for handle 0x3, method 4 on domain 3 (sc 0x4020200) (errno Operation not permitted)
#4

Our execution environment can be downloaded here(http://43.138.42.59:32666/dsp.zip), including model, so libraries and launcher_android.
Our execution steps here:

adb push dsp /data/local/tmp
adb shell
cd /data/local/tmp/dsp
source set_path.sh
./launcher_android --in_config linear_input.json --out_config output.json
#5

In fact, we want to let the model inference on adsp, but it seems that hxv is used when AOT uses hexagon as the target according to

aot_target = "llvm -keys=hexagon -mattr=+hvxv69,+hvx-length128b,+hvx-qfloat,-hvx-ieee-fp -mcpu=hexagonv69 -mtriple=hexagon"
aot_host_target = aot_target

which is only supported on cdsp? Is this optional? Could you give some advice about inference model on adsp?

#6

Sorry, forgot about this one. We are currently assuming CDSP with HVX.

#7

I meet the same problem. I solve this by AoT executor