@masahi @csullivan @Lunderberg @tqchen @yzhliu Hi, I have a tflite model I’d like to run via TVM on Android using tvm_rpc with Vulkan Backend. My host tvm build is compiling with vulkan enabled but the target android runtime isn’t compiling with vulkan.
Using
mod, params = relay.frontend.from_tflite(
tflite_model, shape_dict={input_tensor: input_shape}, dtype_dict={input_tensor: input_dtype}
)
test_target = "vulkan"
arch = "arm64"
target = tvm.target.Target("llvm -mtriple=arm64-linux-android")
target = tvm.target.Target(test_target, host=target)
with tvm.transform.PassContext(opt_level=3):
lib = relay.build(mod, target=target, params=params)
fcompile = ndk.create_shared if run_on_device else None
lib.export_library(lib_fname, fcompile)
The relay is built but when running using:
import tvm
import numpy as np
from tvm import te
from tvm.contrib import graph_executor as runtime
# ctx = remote.cl(0)
#ctx = remote.cpu(0)
ctx = remote.vulkan(0)
print("ctx:", ctx)
# Transfer the model lib to remote device
remote.upload(lib_fname)
# Load the remote module
rlib = remote.load_module(lib_fname)
# Create a runtime executor module
module = runtime.GraphModule(rlib["default"](ctx))
# Run
module.run()
# Benchmark the performance
ftime = module.module.time_evaluator("run", ctx, number=1, repeat=10)
prof_res = np.array(ftime().results) * 1000
print("Mean inference time (std dev): %.2f ms (%.2f ms)" % (np.mean(prof_res), np.std(prof_res)))
I’m getting this error:
RPCError: Error caught from RPC call: [15:58:19] /home/tvm-code/abhinav/clones/open_src/vulkan_ostvm/tvm/src/runtime/library_module.cc:122: Binary was created using {vulkan} but a loader of that name is not registered. Available loaders are const_loader, GraphExecutorFactory, static_library, relax.Executable, AotExecutorFactory, opencl, metadata_module, VMExecutable, metadata, GraphRuntimeFactory. Perhaps you need to recompile with this runtime enabled.
Below is the configuration for host and target build. Host Build:
mkdir -p build
cd build
cp ../cmake/config.cmake .
echo set\(USE_RPC ON\) >> config.cmake
echo set\(USE_GRAPH_EXECUTOR ON\) >> config.cmake
echo set\(USE_LIBBACKTRACE AUTO\) >> config.cmake
echo set\(USE_LLVM llvm-config-10\) >> config.cmake
echo set\(USE_VULKAN ON\) >> config.cmake
cmake ..
make
Got these logs in output:
-- Vulkan_INCLUDE_DIRS=/usr/include/usr/include/spirv-tools/usr/include/spirv/unified1/usr/include/spirv/unified1
-- Vulkan_LIBRARY=/usr/lib/x86_64-linux-gnu/libvulkan.so
-- Vulkan_SPIRV_TOOLS_LIBRARY=/usr/lib/x86_64-linux-gnu/libSPIRV-Tools.a
So the host build is finding vulkan libraries and tools from root and is building with vulkan enabled.
Android Target Build:
Env Variables:
export TVM_ROOT=/home/tvm-code/abhinav/clones/open_src/vulkan_ostvm/tvm/
export TVM_HOME=/home/tvm-code/abhinav/clones/open_src/vulkan_ostvm/tvm/
export ANDROID_NDK_ROOT=/home/tvm-code/Android/Ndk/android-ndk-r25c/
export ANDROID_NDK_HOME=$ANDROID_NDK_ROOT
export NDK_ROOT=$ANDROID_NDK_ROOT
export ANDROID_NDK=$ANDROID_NDK_ROOT
export PYTHONPATH=$TVM_ROOT/python:$PYTHONPATH
export TVM_NDK_CC=$ANDROID_NDK_ROOT/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android28-clang++
export Vulkan_LIBRARY=/home/tvm-code/Android/Ndk/android-ndk-r25c/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/lib/aarch64-linux-android/28/libvulkan.so
mkdir -p build-test
cd build-test
cp ../cmake/config.cmake .
echo set\(USE_OPENCL ON\) >> config.cmake
echo set\(USE_LLVM llvm-config-10\) >> config.cmake
echo set\(USE_RPC ON\) >> config.cmake
echo set\(USE_CPP_RPC ON\) >> config.cmake
echo set\(USE_CPP_RTVM ON\) >> config.cmake
echo set\(USE_SORT ON\) >> config.cmake
echo set\(USE_VULKAN ON\) >> config.cmake
echo set\(USE_GRAPH_EXECUTOR ON\) >> config.cmake
echo set\(USE_LIBBACKTRACE AUTO\) >> config.cmake
echo set\(USE_KALLOC_ALIGNMENT 32\) >> config.cmake
echo set\(ANDROID_ABI arm64-v8a\) >> config.cmake
echo set\(ANDROID_PLATFORM android-28\) >> config.cmake
echo set\(MACHINE_NAME aarch64-linux-gnu\) >> config.cmake
cmake -DCMAKE_TOOLCHAIN_FILE="${ANDROID_NDK_HOME}/build/cmake/android.toolchain.cmake" \
-DANDROID_ABI=arm64-v8a \
-DANDROID_PLATFORM=android-28 \
-DUSE_VULKAN=ON \
-DANDROID_NATIVE_API_LEVEL=23 ..
make tvm_runtime tvm_rpc rtvm
After this I tried
readelf -a libtvm_runtime.so | grep vulkan
and it outputs nothing.