TVM_NDK_CC, target settings for arch=ARMv8-A, part=cortex-A53

rankyung · June 26, 2018, 6:11pm

Hello tvm community,

Can you advise me about TVM_NDK_CC and target settings for arch=ARMv8-A, part=cortex-A53?

I would like to cross-compile TVM-based MXNet model for an android device based on arch=ARMv8-A, part=cortex-A53.
I’ve been stuck in “loading model” for quite long time.
==> Module modelLib = Module.load(libCacheFilePath);

First of all, should I create *.o using this command <-- lib.save(path_o)?
or should I create ***.so using this command <-- lib.export_library(path_so, ndk.create_shared)?

Could you advise on how to set there options - TVM_NDK_CC and target?
I have used several combinations for these.

For TVM_NDK_CC, I tried:
aarch64-linux-android-clang
aarch64-linux-android-gcc
arm-linux-androideabi-clang
arm-linux-androideabi-gcc
I used the following command to create aarch64 toolchains (for aarch64-).
./make-standalone-toolchain.sh --toolchain=aarch64-linux-android-4.9 --platform=android-22 --use-llvm
I used the following command to create arm (32bits) toolchains (for arm-),
./make-standalone-toolchain.sh --platform=android-22 --use-llvm --arch=arm

For target, I tried:
target = ‘llvm -target=armv8a-arm-linux-android -mfloat-abi=soft -mcpu=cortex-a53’
target = ‘llvm -target=armv8a-arm-none-eabi -mfloat-abi=soft -mcpu=cortex-a53’
target = ‘llvm -target=armv7-none-linux-androideabi -mfloat-abi=soft -mcpu=cortex-a53’
target = ‘llvm -target=armv7-none-linux-androideabi -mfloat-abi=soft’
and more…

eqy · June 26, 2018, 6:49pm

Have you tried TVM_NDK_CC=aarch64-linux-android-g++? (as suggested in the docs )

The example android test script uses a target host of llvm -target=arm64=linux-android

I would also check that you are linking in the required libraries (e.g., if you are using OpenCL) at APP build time.

rankyung · June 26, 2018, 7:36pm

Yes, I have tried to TVM_NDK_CC=aarch64-linux-android-g++ and llvm -target=arm64-linux-android. However, it returns the same error below.

A/libc: /Volumes/Android/buildbot/src/android/ndk-release-r17/external/libcxx/…/…/external/libcxxabi/src/abort_message.cpp:73: abort_message: assertion “terminating with uncaught exception of type dmlc::Error: [19:31:39] ~/gitRepos/tvm/apps/android_rpc/app/src/main/jni/jni_helper_func.h:182: Do NOT know how to handle return type code -141177368” failed
A/libc: Fatal signal 6 (SIGABRT), code -6 in tid 9172 (AsyncTask #1)

I don’t use any external libraries.
This is the config.mk
APP_ABI = all
APP_PLATFORM = android-22
USE_OPENCL = 0
ADD_C_INCLUDES =
ADD_LDLIBS =

When i set APP-ABI = arm64-v8a,
it doesn’t work.
APP_ABI = armeabi-v7a
It works.

Are there any other suggestions?

eqy · June 26, 2018, 7:46pm

Is your android device running a 32-bit or 64-bit OS? That should inform whether you should use v7a or v8a. If you create an arm64 standalone toolchain when your OS/CPU is 32-bit, then that will likely be problematic.

For an example on the export library and module load, see the android_rpc_test.py script.

rankyung · June 26, 2018, 8:09pm

I got a working combination.

cross-compiler: arm-linux-androideabi-g++
it is made by using this command.
cd ~/android-ndk-r17b/build/tools/
./make-standalone-toolchain.sh --platform=android-22 --use-llvm --arch=arm --install-dir=****
That is, TVM_NDK_CC=~/arm-linux-toolchain/bin/arm-linux-androideabi-g++
config.mk
APP_ABI = armeabi-v7a
APP_PLATFORM = android-22
USE_OPENCL = 0
ADD_C_INCLUDES =
ADD_LDLIBS =
target = ‘llvm -target=armv7a-none-linux-android -mfloat-abi=soft -mcpu=cortex-a53’

@eqy,
When I executed the following command in my android device,
getprop ro.product.cpu.abi
I got this result
arm64-v8a

In addition, when i executed
cat /proc/cpuinfo
I have
Processor : AArch64 Processor rev 3 (aarch64)
processor : 0
model name : AArch64 Processor rev 3 (aarch64)
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd03
CPU revision : 3

It seems the device running a 64bit OS, doesn’t it?
Thus, well… I don’t know why the above combination works… XD

eqy · June 26, 2018, 8:13pm

That’s interesting, can you check explicitly if the Android version is 32/64bit? e.g., http://techknowzone.com/how-to-check-if-your-android-device-is-running-on-32-bit-or-64-bit-android-os/

rankyung · June 26, 2018, 8:21pm

@eqy

Unfortunately, my android device (echo dot) doesn’t have a display unlike smartphone.
So i cannot search the info by the way stated in the webpage.
That is why, i used getprop ro.product.cpu.abi, and cat /proc/cpuinfo.
Please let me know if you know any other commands that i can directly get the 32/64bit information.

eqy · June 26, 2018, 8:23pm

If you have a cli and it behaves somewhat like a GNU/Linux box, can you try uname -m?

rankyung · June 26, 2018, 8:28pm

uname -m returns
/system/bin/sh: uname: not found

Some commands works, others doesn’t…

eqy · June 26, 2018, 8:46pm

I guess one hacky way is to check the memory mapping of a random process in /proc/$$/maps (just pick any number, or just type $$ to match a number) https://unix.stackexchange.com/questions/136407/is-my-linux-arm-32-or-64-bit#comment305314_136519.

Can you post some address ranges from a random process there?

rankyung · June 26, 2018, 9:11pm

Yes sure, this is the result of cat /proc/self/maps

root@biscuit:/data # cat /proc/self/maps
7fb6148000-7fb614b000 r-xp 00000000 b3:0d 4326 7fb614b000-7fb615a000 —p 00000000 00:00 7fb615a000-7fb615b000 r–p 00002000 b3:0d 4326 7fb615b000-7fb615c000 rw-p 00003000 b3:0d 4326 7fb615c000-7fb6187000 r-xp 00000000 b3:0d 4249 7fb6187000-7fb6197000 —p 00000000 00:00 7fb6197000-7fb6198000 r–p 0002b000 b3:0d 4249 7fb6198000-7fb6199000 rw-p 0002c000 b3:0d 4249 7fb6199000-7fb619b000 r-xp 00000000 b3:0d 4270 7fb619b000-7fb61aa000 —p 00000000 00:00 7fb61aa000-7fb61ab000 r–p 00001000 b3:0d 4270 7fb61ab000-7fb61ac000 rw-p 00002000 b3:0d 4270 7fb61ac000-7fb61cc000 r–s 00000000 00:0d 41 7fb61cc000-7fb6255000 r-xp 00000000 b3:0d 4154 7fb6255000-7fb6265000 —p 00000000 00:00 7fb6265000-7fb6269000 r–p 00089000 b3:0d 4154 7fb6269000-7fb626c000 rw-p 0008d000 b3:0d 4154 7fb626c000-7fb627a000 rw-p 00000000 00:00 7fb627a000-7fb628c000 r-xp 00000000 b3:0d 217 7fb629b000-7fb629c000 r–p 00011000 b3:0d 217 7fb629c000-7fb629d000 rw-p 00012000 b3:0d 217 7fb629d000-7fb629e000 rw-p 00000000 00:00 7fb629e000-7fb62e2000 r-xp 00000000 b3:0d 324 7fb62e8000-7fb62e9000 r–p 00000000 7fb62e9000-7fb62ea000 r–p 00000000 00:00 0 7fb62ea000-7fb62eb000 rw-p 00000000 00:00 0 7fb62eb000-7fb62ec000 r–p 00000000 00:00 0 7fb62ec000-7fb62ed000 r–p 00000000 7fb62ed000-7fb62ef000 rw-p 00000000 00:00 7fb62ef000-7fb62f0000 r–p 00000000 00:00 0 7fb62f0000-7fb62f1000 r-xp 00000000 00:00 0 7fb62f1000-7fb62f3000 r–p 00043000 b3:0d 324 7fb62f3000-7fb62f4000 rw-p 00045000 b3:0d 324 7fb62f4000-7fb62f5000 rw-p 00000000 00:00 7fcf4ef000-7fcf508000 rw-p 00000000 00:00 0 7fe2221000-7fe2a20000 rw-p 00000000 00:00 0 /system/lib64/libstdc++.so
0
/system/lib64/libstdc++.so
/system/lib64/libstdc++.so
/system/lib64/libm.so
0
/system/lib64/libm.so
/system/lib64/libm.so
/system/lib64/libnetd_client.so
0
/system/lib64/libnetd_client.so
/system/lib64/libnetd_client.so
/dev/properties
/system/lib64/libc.so
0
/system/lib64/libc.so
/system/lib64/libc.so
0
/system/bin/linker64
/system/bin/linker64
/system/bin/linker64
0
/system/bin/sh
00:00 0
[anon:linker_alloc]
[anon:linker_alloc]
[anon:linker_alloc]
00:00 0
0
[vvar]
[vdso]
/system/bin/sh
/system/bin/sh
0
[heap]
[stack]

rankyung · June 26, 2018, 9:12pm

It looks like 64bits device

eqy · June 26, 2018, 9:14pm

Interesting, well then in theory the 64bit targets should also work.

rankyung · June 26, 2018, 9:17pm

Yea it’s interesting, I will re-try with 64bits combinations, and will post the result.
Thank you so much @eqy

rankyung · June 26, 2018, 11:38pm

I could make .so file with the following cross-compiler and target options

cross-compiler: aarch64-linux-android-g++
it is made by using this command.
cd ~/android-ndk-r17b/build/tools/
./make-standalone-toolchain.sh --platform=android-22 --use-llvm --arch=arm64 --install-dir=~/arm64-toolchain

That is, TVM_NDK_CC=~/arm64-toolchain/bin/aarch64-linux-android-g++
target = ‘llvm -target=aarch64-none-linux-android -mfloat-abi=soft -mcpu=cortex-a53’

However, I couldn’t make an android app with APP_ABI = arm64-v8a in config.mk
APP_ABI = all also doesn’t work

welcome for further suggestions or comments.

dayanandasiet · June 28, 2018, 6:41am

Dear Rankyung,

Follow the below steps to generate toolchian and try to generate application with below export command

Tool chain generate with below instruction
./make-standalone-toolchain.sh --platform=android-24 --use-llvm --arch=arm64 --install-dir=/home/user/software/android-toolchain-arm64/
Download Java and SDK, set proper path
export TVM_NDK_CC=/home/user/software/android-toolchain-arm64/bin/aarch64-linux-android-g++
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export ANDROID_HOME=/home/user/software/android-sdk-linux
build mxnet model with nnvm with below config/parameter and use same library, param and graph on your android application

target = “llvm -target=arm64-linux-android”
target_host = None
reference mobile_darknet_save.py
Compile application android deploy using this config.mk configuration for CPU flavor

rankyung · June 28, 2018, 5:19pm

Hi @dayanandasiet,

Thanks for the direction.
I successfully generated library, param and graph using the cross-compiler (aarch64-linux-android-g++) and target option (target = “llvm -target=arm64-linux-android”).
However, when I use “APP_ABI = arm64-v8a” in config.mk, it returns an error below.
Failure [INSTALL_FAILED_NO_MATCHING_ABIS]

Interestingly, when I build mxnet model with

32bit-based cross-compiler (arm-linux-androideabi-g++)
target = ‘llvm -target=armv7a-none-linux-android -mfloat-abi=soft -mcpu=cortex-a53’

and run it with “APP_ABI=armeabi-v7a”,
it works.

dayanandasiet · June 29, 2018, 4:27pm

Depend on target hardware and software version need to select APP_ABI version. Latest most hardware 64bit arm, so select right APP_ABI mode to compile your application.

Thanks