Cross compile the runtime only for aarch64

[mherkazandjian]

Hi,

I am trying to compile the runtime only for aarch64. I would like to use the runtime on a aarch64 qemu vm (for the time being before testing on the actual hardware - for this “question” the target device is irrelevant since i want to cross compile (on a x86 host) the runtime for aarch64.

In this thread https://discuss.tvm.apache.org/t/why-do-we-need-to-build-tvm-runtime-on-remote-device-for-auto-tvm/2866/10 it is suggested to use a bunch of flags at the cmake level. The compiler that i am using is aarch64-linux-gnu-gcc-8 and I am working in a docker container that is an instance of the following image https://github.com/apache/tvm/blob/main/docker/Dockerfile.ci_cpu.

Inside the container, i clone the tvm source code and i execute

mkdir build cd build cp …/cmake/config.cmake . cmake …
-DCMAKE_SYSTEM_NAME=Linux
-DCMAKE_SYSTEM_VERSION=1
-DCMAKE_C_COMPILER=/usr/bin/aarch64-linux-gnu-gcc-8
-DCMAKE_CXX_COMPILER=/usr/bin/aarch64-linux-gnu-g+±8
-DCMAKE_FIND_ROOT_PATH=/usr/aarch64-linux-gnu
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER
-DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY

all goes well until the linking part

It looks like there is an issue linking with libbacktrace, any clue why this is happening? maybe some flags should be passed when compiling libbacktrace?

as a workaround, i did the following to disable linking against libbacktrace to disable linking against libbacktrace (i am not sure what consequences this has, but the runtime .so was produced and i did not get any errors:

cmake …
-DCMAKE_SYSTEM_NAME=Linux
-DCMAKE_SYSTEM_VERSION=1
-DCMAKE_C_COMPILER=/usr/bin/aarch64-linux-gnu-gcc-8
-DCMAKE_CXX_COMPILER=/usr/bin/aarch64-linux-gnu-g+±8
-DCMAKE_FIND_ROOT_PATH=/usr/aarch64-linux-gnu
-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER
-DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY
-DCMAKE_C_FLAGS="-DTVM_USE_LIBBACKTRACE=0" \
-DCMAKE_CXX_FLAGS="-DTVM_USE_LIBBACKTRACE=0"

are there up to date instructions on cross compiling the runtime for aarch64 (and riscv [didn’t try that yet])

tnx,

[junrushao]

we could turn USE_LIBBACKTRACE to OFF in config.cmake so that it uses legacy behavior.

On the particular linking issue, it is a possible issue that libbacktrace, compiled with GNU make, is not using the same toolchain as cmake is using, so what about we set environment variables CC, CXX, etc, properly so that GNU make could select the right toolchain?

[mherkazandjian]

i tried to cross compile for risv64 and had the same issue with linking libbacktrace. Setting “USE_LIBBACKTRACE OFF” in the config.cmake file indeed solved the issue without having to specify the -DCMAKE_C_FLAGS and -DCMAKE_CXX_FLAGS

i’ll check using CC and CXX.

[mherkazandjian]

using

CC=/usr/bin/aarch64-linux-gnu-gcc-8 CXX=/usr/bin/aarch64-linux-gnu-g+±8

failed to configure libbacktrace since the “./configure” step did not build a test program using “CC”. Using e.g CC=/usr/bin/aarch64-linux-gnu-gcc-8 on a x86 host would build a test executable and when executed it fails because it was built using a binary for a aarch64 host. The right way to configure libbacktrace would be to somehow pass --host=/usr/aarch64-linux-gnu to the “./configure” step of libbacktrace through the cmake config of tvm. I tried this in standalone mode for libbacktrace and libbacktrace was built successfully. this looks like a bug actually, since one should expect that if at the cmake level CC is set to something, it should be used to all the sub-components.

[junrushao]

It is definitely interesting finding, so there might be some misconfiguration in our cmake external project, where the cmake flags is not properly passed to the external project (libbacktrace).

Would you like to tweak this file, and adding the toolchain as suggested in this post? Thank you for the valuable time!

[mherkazandjian]

ok, i’ll give it a shot.

i have a question though about libbacktrace. I suppose it is not necssary for the runtime in an embedded device, right? so for the time being, just turning it off would be good enough to get things running. (i wonder what other do when it comes to cross compiling the runtime).

[junrushao]

Right. It is not a hard dependency of the system, so on an embedded device, it is safe to just turn it off

[mherkazandjian]

Great. Tnx for the feedback. i’ll work on the cmake fixes in the coming few days and move the discussoin to a PR on github.

[mherkazandjian]

i think i fixed it https://github.com/apache/tvm/pull/7917

[junrushao]

Thanks for the fix PR!