Can not run the model with Vulkan backend on Android

WildTaras · May 26, 2021, 10:35am

Hello everyone, I have a network in form of Pytorch model and I’d like to try using it via TVM in my Android application. At first, I have compiled my model with CPU target:

target = tvm.target.Target(“llvm -mtriple=%s-linux-android” % arch)

CPU model was compiled successfuly and works as expected in my Application. So now I have decided to try building the model with Vulkan backend because I’m interested to get the best possible inference time. I have compiled my model with target

target = tvm.target.Target(“llvm -mtriple=%s-linux-android” % arch)
target = tvm.target.Target(“vulkan”, host=target)

Also I have built TVM runtime with Vulkan extension and it seems like that the process was successful. Unfortunately, when I try to use my Vulkan model+runtime batch I have an error:

E/libc++abi: terminating with uncaught exception of type tvm::runtime::InternalError: [13:07:47] /workspace/tvm_android/src/runtime/library_module.cc:116: Binary was created using vulkan but a loader of that name is not registered. Available loaders are GraphRuntimeFactory, GraphExecutorFactory, VMExecutable, metadata. Perhaps you need to recompile with this runtime enabled.
Stack trace not available when DMLC_LOG_STACK_TRACE is disabled at compile time.

Meanwhile I’m digging TVM source code to find an answer, I’d be happy if someone can point me to my mistake or give a valuable advice. Just in case, I’m attaching my inference code below:

Device dev {kDLVulkan, device_id};
m_mod = tvm::runtime::Module::LoadFromFile(modelName);

Module def = m_mod.GetFunction("default")(dev);
m_inputFunc = def.GetFunction("set_input");
m_runFunc = def.GetFunction("run");
m_outputFunc = def.GetFunction("get_output");

m_input = NDArray::Empty({1, 3, 256, 256}, DLDataType{kDLFloat, 32, 1}, dev);
m_output = NDArray::Empty({1, 209, 64, 64}, DLDataType{kDLFloat, 32, 1}, dev);

masahi · May 26, 2021, 11:39am

Have you looked at tvm/apps/android_rpc at main · apache/tvm · GitHub and ran tvm/android_rpc_test.py at main · apache/tvm · GitHub? I’ve never worked with android phone, but this is what I know to get started with TVM + android + vk.

We improved our vulkan backend quite a bit recently, it works perfectly on AMD and ok-ish on NV (some models fail to compute correct outputs). I’m very interested in mobile GPU result. What is your GPU?

WildTaras · May 26, 2021, 12:28pm

I use Samsung S9 with Qualcomm CPU and Adreno GPU

Concerning rpc tracker, when I try to run android_rpc_test.py, I can connect to the tracker, but cannot got request from the phone. I have checked the availability device via this request

python3 -m tvm.exec.query_rpc_tracker --host=0.0.0.0 --port=9190

It showed me, that smartphone available, but unfortunately no respond from it via android_rpc_test.py, so I decide to deploy model and run it on android device directly

masahi · May 26, 2021, 12:52pm

Have you built your tvm runtime on the device with vulkan support? The error message suggests the runtime on the device does not have vulkan enabled.

WildTaras · May 26, 2021, 1:23pm

Yes, I ran this command

cmake .. -DCMAKE_TOOLCHAIN_FILE=$ANDROID_NDK_HOME/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_NATIVE_API_LEVEL=23 -DANDROID_ARM_NEON=ON -DUSE_VULKAN=ON -DUSE_GRAPH_EXECUTOR=ON

And on the next step, I use it:

make runtime

And got the next log:

Scanning dependencies of target tvm_runtime_objs
[  0%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/builtin_fp16.cc.o
[  0%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/c_runtime_api.cc.o
[  0%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/container.cc.o
[  9%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/cpu_device_api.cc.o
[  9%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/dso_library.cc.o
[  9%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/file_utils.cc.o
[  9%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/library_module.cc.o
[ 18%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/logging.cc.o
[ 18%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/metadata_module.cc.o
[ 18%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/module.cc.o
[ 18%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/ndarray.cc.o
[ 27%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/object.cc.o
[ 27%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/profiling.cc.o
[ 27%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/registry.cc.o
[ 27%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/system_library.cc.o
[ 36%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/thread_pool.cc.o
[ 36%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/threading_backend.cc.o
[ 36%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/vm/bytecode.cc.o
[ 45%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/vm/executable.cc.o
[ 45%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/vm/memory_manager.cc.o
[ 45%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/vm/vm.cc.o
[ 45%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/workspace_pool.cc.o
[ 54%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/rpc/rpc_channel.cc.o
[ 54%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/rpc/rpc_device_api.cc.o
[ 54%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/rpc/rpc_endpoint.cc.o
[ 54%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/rpc/rpc_event_impl.cc.o
[ 63%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/rpc/rpc_local_session.cc.o
[ 63%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/rpc/rpc_module.cc.o
[ 63%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/rpc/rpc_pipe_impl.cc.o
[ 63%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/rpc/rpc_server_env.cc.o
[ 72%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/rpc/rpc_session.cc.o
[ 72%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/rpc/rpc_socket_impl.cc.o
[ 72%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/graph_executor/graph_executor.cc.o
[ 72%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/graph_executor/graph_executor_factory.cc.o
[ 81%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/graph_executor/debug/graph_executor_debug.cc.o
[ 81%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/vm/profiler/vm.cc.o
[ 81%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/contrib/random/random.cc.o
[ 90%] Building CXX object CMakeFiles/tvm_runtime_objs.dir/src/runtime/contrib/sort/sort.cc.o
[ 90%] Built target tvm_runtime_objs
Scanning dependencies of target tvm_runtime
[100%] Linking CXX shared library libtvm_runtime.so
[100%] Built target tvm_runtime
Scanning dependencies of target runtime
[100%] Built target runtime

I didn’t see any log about building Vulkan.cc.o, but I am not sure is it necessary in this case

masahi · May 26, 2021, 7:51pm

I’d suggest revisiting the RPC way, since this is the standard way to target Android I believe. I’m tagging people who might be able to help on this. @csullivan @Lunderberg @tqchen @yzhliu

WildTaras · May 26, 2021, 9:02pm

Thank you very much, I really appreciate any attempting to help.

WildTaras · May 28, 2021, 10:11am

Helo @masahi. I tried to compile runtime explicitly, where I put the path to the Vulkan SDK like this /Users/user/VulkanSDK/1.2.176.1/macOS Unfortunately, It gets me output

    -- Custom Vulkan SDK PATH=/Users/user/VulkanSDK/1.2.176.1/macOS
    CMake Error at cmake/modules/Vulkan.cmake:23 (message):
      Cannot find Vulkan, USE_VULKAN=/Users/user/VulkanSDK/1.2.176.1/macOS
    Call Stack (most recent call first):
      CMakeLists.txt:372 (include)
    -- Configuring incomplete, errors occurred!
    See also "/Users/user/Documents/AlgofaceWork/TVM/tvm/build/CMakeFiles/CMakeOutput.log".

Also, I have export all necessary Vulkan paths, like VULKAN_SDK, VK_LAYER_PATH etc renew PATH, and in config.cmake I just set ON in set(Vulkan ON), but it also didnt work and gave me the error seems like above.

Could you please help me?

Lunderberg · May 28, 2021, 3:14pm

What are the contents of the /Users/user/VulkanSDK/1.2.176.1/macOS directory? I haven’t compiled on maxOS before, but I think it VULKAN_SDK should be set such that the following files can be found. Do these files exist with your current value of VULKAN_SDK?

${VULKAN_SDK}/include/vulkan/vulkan.h
${VULKAN_SDK}/lib/libvulkan.so
${VULKAN_SDK}/lib/libvulkan-1.so

WildTaras · May 28, 2021, 6:34pm

Hello @Lunderberg. Thank you for your response. I compiled tmv on macOS and Ubuntu. In both cases necessary libraries were existed. In Linux case there are: libvulkan.so and libvulkan-1.so In MacOS case there are: libvulkan.dylib and libvulkan-1.dylib But, I got the same error in both ways. What is more, I try to compile tmv runtime, using this command (I edited config.cmake and set Vulkan to ON):

cmake -DUSE_LLVM=llvm-config-8 \
      -DUSE_RPC=ON \
      -DUSE_SORT=ON \
      -DUSE_VULKAN=ON \
      -DUSE_GRAPH_EXECUTOR=ON \
      ..

And got the next result:

CMake Error in CMakeLists.txt:
  Found relative path while evaluating include directories of "tvm_runtime":

    "_spirv-NOTFOUND"

Maybe, it can gives additional information to help find the way of resolving problem.

Lunderberg · May 28, 2021, 7:03pm

Thank you, that looks like it’s not finding the SPIRV-tools library. When you run cmake, what does it print out for Vulkan_SPIRV_TOOLS_LIBRARY?

WildTaras · May 28, 2021, 7:08pm

I decide to share with you all logs. Concerning your questions:

Vulkan_SPIRV_TOOLS_LIBRARY=/usr/local/lib/libSPIRV-Tools.a

And, as I said, all logs below:

-- Build with RPC support...
-- Build with Graph Executor support...
-- Build with profiler...
-- VTA build with VTA_HW_PATH=/Users/user/Documents/AlgofaceWork/TVM/tvm/3rdparty/vta-hw
-- Build VTA runtime with target: sim
--     Vulkan_INCLUDE_DIRS=/Users/user/VulkanSDK/1.2.176.1/macOS/include/Users/user/VulkanSDK/1.2.176.1/macOS/include/spirv-tools_spirv-NOTFOUND_glsl_std-NOTFOUND
    -- Vulkan_LIBRARY=/usr/local/lib/libvulkan.dylib
    -- Vulkan_SPIRV_TOOLS_LIBRARY=/usr/local/lib/libSPIRV-Tools.a
    -- Build with Vulkan support
    -- Build with contrib.random
    -- Build with contrib.sort
    -- Build with contrib.hybriddump
    -- Git found: /usr/local/bin/git
    -- Found TVM_GIT_COMMIT_HASH=2cde3dc0e7047ab948dcf6970d0fa488f47b1057
    -- Autoset: USE_LIBBACKTRACE=OFF in Darwin
    -- Building with TVM Map...
    -- Build with thread support...
    -- Configuring done
    CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
    Please set them or make sure they are set and tested correctly in the CMake files:
    _glsl_std
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
    _spirv
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm
       used as include directory in directory /Users/user/Documents/AlgofaceWork/TVM/tvm

    CMake Error in CMakeLists.txt:
      Found relative path while evaluating include directories of "tvm_runtime":

        "_spirv-NOTFOUND"



    CMake Error in CMakeLists.txt:
      Found relative path while evaluating include directories of "tvm_runtime":

        "_spirv-NOTFOUND"



    CMake Error in CMakeLists.txt:
      Found relative path while evaluating include directories of "tvm":

        "_spirv-NOTFOUND"



    CMake Error in CMakeLists.txt:
      Found relative path while evaluating include directories of "tvm":

        "_spirv-NOTFOUND"



    CMake Error in CMakeLists.txt:
      Found relative path while evaluating include directories of
      "tvm_runtime_objs":

        "_spirv-NOTFOUND"



    CMake Error in CMakeLists.txt:
      Found relative path while evaluating include directories of
      "tvm_runtime_objs":

        "_spirv-NOTFOUND"



    CMake Error in CMakeLists.txt:
      Found relative path while evaluating include directories of
      "tvm_runtime_objs":

        "_spirv-NOTFOUND"



    CMake Error in CMakeLists.txt:

    Taras Khalep, [28.05.21 22:05]
    Found relative path while evaluating include directories of "tvm_objs":

    "_spirv-NOTFOUND"

WildTaras · May 31, 2021, 6:34pm

Hello @Lunderberg. I have resolved the problem with the linking to the different part of vulkan library. But I faced the problem, which I described below during the runtime compilation for Android device. If I try to compile tvm_runtime on Ubuntu with Vulkan, I got next output:

Built target tvm_runtime_objs
make[3]: *** No rule to make target `/usr/local/lib/libvulkan.so', needed by `libtvm_runtime.so'.  Stop.

And If I do the same compilation process but on the MacOS, I got next error:

/usr/local/lib/libvulkan.dylib: file not recognized: File format not recognized
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

I appreciate any hints concerning the way how to solve this problem.

WildTaras · June 1, 2021, 8:30am

Hello @Lunderberg. I have faced new problem. Could you please take a look on the log on the end of the compilation tvm_runtime

[100%] Built target tvm_runtime_objs
[100%] Linking CXX shared library libtvm_runtime.so
/home/whoami/vulkan/1.2.170.0/x86_64/lib/libvulkan.so: error adding symbols: File in wrong format
clang++: error: linker command failed with exit code 1 (use -v to see invocation)

Lunderberg · June 1, 2021, 3:56pm

For this one, this looks like it isn’t finding the vulkan library when compiling. Since it is still included as a dependency in the makefile that cmake outputs, make thinks that it is supposed to create libvulkan.so, but doesn’t know how. I would have expected this to cause an issue earlier when running cmake, since the Vulkan_LIBRARY variable should require finding the correct location of libvulkan.so. After the changes you made to point to a different location, what did the cmake output look like?

The errors on both the Ubuntu and MacOS sides look like it is trying to link together object files targeting different architectures. From the path, the libvulkan.so you’re using is a 64-bit version. What does running uname -a output on your system, and is there any chance you’re using a 32-bit operating system?

Similar question here, what is the architecture of the MacOS system (x86_64 or M1), and what architecture does libvulkan.dylib use? There may be another mismatch there.

@csullivan When you compiled libtvm_runtime.so to run an rpc server on an Android device, were there any special changes you needed to make for it? Could you share the config.cmake file that you used in order to do so?

WildTaras · June 2, 2021, 5:23am

@Lunderberg Thank you very much for your detailed response. I have MacBook with Intel Core I7 on the board There are outputs of the command uname -a

MacBook:

Darwin MacBook-Pro-Mojo.local 19.5.0 Darwin Kernel Version 19.5.0: Tue May 26 20:41:44 PDT 2020; root:xnu-6153.121.2~2/RELEASE_X86_64 x86_64

Ubuntu:

Linux whoami-IP 5.8.0-53-generic #60~20.04.1-Ubuntu SMP Thu May 6 09:52:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

I made deep investigation and did the next movements:

Refactored Vulkan.cmake and FindVulkan.cmake. It means that I set all paths to the files for linking vulkan into tvm_runtime lib in a correct way. In my case it looks like:

include_directories(/Users/user/VulkanSDK/1.2.176.1/macOS/include/)
list(APPEND TVM_LINKER_LIBS /Users/user/VulkanSDK/1.2.176.1/macOS/lib/libSPIRV-Tools.a)
list(APPEND TVM_RUNTIME_LINKER_LIBS /Users/user/VulkanSDK/1.2.176.1/macOS/lib/libvulkan.so)

It must work, because all paths are set correctly, but I got the next error:

/Users/user/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/bin/ld: skipping incompatible /Users/user/VulkanSDK/1.2.176.1/macOS/lib/libvulkan.so when searching for -lvulkan

/Users/user/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/lib/gcc/aarch64-linux-android/4.9.x/../../../../aarch64-linux-android/bin/ld: cannot find -lvulkan

I saw, that Cmakelists of tvm library try to link path of the libvulkan.so to tvm_runtime, I put the line of the code below:

target_link_libraries(tvm_runtime 
                      PRIVATE 
                      ${TVM_RUNTIME_LINKER_LIBS})

After that, I saw how original cmake implements FindVulkan.cmake. The main idea that it creates the module besides creating the variables with the paths. Sample of code I put below:

if(Vulkan_FOUND AND NOT TARGET Vulkan::Vulkan) add_library(Vulkan::Vulkan UNKNOWN IMPORTED) set_target_properties(Vulkan::Vulkan PROPERTIES IMPORTED_LOCATION “${Vulkan_LIBRARIES}” INTERFACE_INCLUDE_DIRECTORIES “${Vulkan_INCLUDE_DIRS}”)

So I try to create the module and link it to the tvm_runtime: I created the library in CmakeLists to make it global scope:

add_library(Vulkan_LIB SHARED IMPORTED)

After that, I link path to libvulkan.so to the Vukan_LIB library inside the Vulkan.cmake:

  set_target_properties(Vulkan_LIB PROPERTIES IMPORTED_LOCATION "${Vulkan_LIBRARY}"
                                    INTERFACE_INCLUDE_DIRECTORIES "/Users/user/VulkanSDK/1.2.176.1/macOS/include/")

And final step, link created Vulkan_LIB to the tvm_runtime inside CmakeLists:

target_link_libraries(tvm_runtime ${Vulkan_LIB})

But it also gives the error like it doesnt see libvulkan.so, because there are no implementation of different functions:

> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::DeleteHostVisibleBuffer(tvm::runtime::vulkan::VulkanHostVisibleBuffer*)':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:76: undefined reference to `vkUnmapMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:76: undefined reference to `vkUnmapMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:79: undefined reference to `vkFreeMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:79: undefined reference to `vkFreeMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:82: undefined reference to `vkDestroyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:82: undefined reference to `vkDestroyBuffer'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::FindMemoryType(tvm::runtime::vulkan::VulkanContext const&, VkBufferCreateInfo, unsigned int)':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:138: undefined reference to `vkCreateBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:138: undefined reference to `vkCreateBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:141: undefined reference to `vkGetBufferMemoryRequirements'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:141: undefined reference to `vkGetBufferMemoryRequirements'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:144: undefined reference to `vkGetPhysicalDeviceMemoryProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:144: undefined reference to `vkGetPhysicalDeviceMemoryProperties'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::CreateBuffer(tvm::runtime::vulkan::VulkanContext const&, unsigned long, unsigned int, unsigned int)':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:175: undefined reference to `vkCreateBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:175: undefined reference to `vkCreateBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:208: undefined reference to `vkAllocateMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:208: undefined reference to `vkAllocateMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:224: undefined reference to `vkBindBufferMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:224: undefined reference to `vkBindBufferMemory'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::VulkanDeviceAPI::GetDeviceDescription(VkInstance_T*, VkPhysicalDevice_T*, std::__ndk1::vector<char const*, std::__ndk1::allocator<char const*> > const&, std::__ndk1::vector<char const*, std::__ndk1::allocator<char const*> > const&)':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:488: undefined reference to `vkGetPhysicalDeviceProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:488: undefined reference to `vkGetPhysicalDeviceProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:541: undefined reference to `vkGetPhysicalDeviceFeatures'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:541: undefined reference to `vkGetPhysicalDeviceFeatures'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:531: undefined reference to `vkGetInstanceProcAddr'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:531: undefined reference to `vkGetInstanceProcAddr'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:631: undefined reference to `tvm::IntImm::IntImm(tvm::runtime::DataType, long, tvm::Span)'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:643: undefined reference to `tvm::Target::Target(tvm::runtime::Map<tvm::runtime::String, tvm::runtime::ObjectRef, void, void> const&)'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `Bool':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/include/tvm/ir/expr.h:329: undefined reference to `tvm::IntImm::IntImm(tvm::runtime::DataType, long, tvm::Span)'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `Integer':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/include/tvm/ir/expr.h:366: undefined reference to `tvm::IntImm::IntImm(tvm::runtime::DataType, long, tvm::Span)'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::operator<<(std::__ndk1::basic_ostream<char, std::__ndk1::char_traits<char> >&, tvm::runtime::ObjectRef const&)':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/include/tvm/node/repr_printer.h:70: undefined reference to `tvm::ReprPrinter::Print(tvm::runtime::ObjectRef const&)'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/include/tvm/node/repr_printer.h:70: undefined reference to `tvm::ReprPrinter::Print(tvm::runtime::ObjectRef const&)'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/include/tvm/node/repr_printer.h:70: undefined reference to `tvm::ReprPrinter::Print(tvm::runtime::ObjectRef const&)'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `operator()':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:723: undefined reference to `vkEnumerateInstanceLayerProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:723: undefined reference to `vkEnumerateInstanceLayerProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:751: undefined reference to `vkEnumerateInstanceExtensionProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:751: undefined reference to `vkEnumerateInstanceExtensionProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:771: undefined reference to `vkGetInstanceProcAddr'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:771: undefined reference to `vkGetInstanceProcAddr'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `VulkanDeviceAPI':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:799: undefined reference to `vkCreateInstance'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:799: undefined reference to `vkCreateInstance'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:803: undefined reference to `vkEnumeratePhysicalDevices'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:803: undefined reference to `vkEnumeratePhysicalDevices'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:825: undefined reference to `vkGetPhysicalDeviceProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:825: undefined reference to `vkGetPhysicalDeviceProperties'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `operator()':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:843: undefined reference to `vkEnumerateDeviceExtensionProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:843: undefined reference to `vkEnumerateDeviceExtensionProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:846: undefined reference to `vkEnumerateDeviceExtensionProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:846: undefined reference to `vkEnumerateDeviceExtensionProperties'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `VulkanDeviceAPI':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:920: undefined reference to `vkCreateDevice'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:920: undefined reference to `vkCreateDevice'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:924: undefined reference to `vkGetDeviceQueue'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:924: undefined reference to `vkGetDeviceQueue'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:941: undefined reference to `vkCreateBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:941: undefined reference to `vkCreateBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:942: undefined reference to `vkGetBufferMemoryRequirements'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:942: undefined reference to `vkGetBufferMemoryRequirements'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:943: undefined reference to `vkDestroyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:943: undefined reference to `vkDestroyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:947: undefined reference to `vkCreateBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:947: undefined reference to `vkCreateBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:948: undefined reference to `vkGetBufferMemoryRequirements'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:948: undefined reference to `vkGetBufferMemoryRequirements'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:949: undefined reference to `vkDestroyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:949: undefined reference to `vkDestroyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:955: undefined reference to `vkGetPhysicalDeviceMemoryProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:955: undefined reference to `vkGetPhysicalDeviceMemoryProperties'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::VulkanDeviceAPI::GetComputeQueueFamilies(VkPhysicalDevice_T*)':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1017: undefined reference to `vkGetPhysicalDeviceQueueFamilyProperties'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1017: undefined reference to `vkGetPhysicalDeviceQueueFamilyProperties'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::DeleteHostVisibleBuffer(tvm::runtime::vulkan::VulkanHostVisibleBuffer*)':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:76: undefined reference to `vkUnmapMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:76: undefined reference to `vkUnmapMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:79: undefined reference to `vkFreeMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:79: undefined reference to `vkFreeMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:82: undefined reference to `vkDestroyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:82: undefined reference to `vkDestroyBuffer'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::GetOrAllocate(int, unsigned long, unsigned int, unsigned int, std::__ndk1::unordered_map<unsigned long, std::__ndk1::unique_ptr<tvm::runtime::vulkan::VulkanHostVisibleBuffer, std::__ndk1::default_delete<tvm::runtime::vulkan::VulkanHostVisibleBuffer> >, std::__ndk1::hash<unsigned long>, std::__ndk1::equal_to<unsigned long>, std::__ndk1::allocator<std::__ndk1::pair<unsigned long const, std::__ndk1::unique_ptr<tvm::runtime::vulkan::VulkanHostVisibleBuffer, std::__ndk1::default_delete<tvm::runtime::vulkan::VulkanHostVisibleBuffer> > > > >*, bool)':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1379: undefined reference to `vkMapMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1379: undefined reference to `vkMapMemory'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::VulkanStream::Synchronize()':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:140: undefined reference to `vkEndCommandBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:140: undefined reference to `vkEndCommandBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:156: undefined reference to `vkQueueSubmit'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:156: undefined reference to `vkQueueSubmit'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `~lock_guard':
> 
> /Users/user/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/c++/v1/__mutex_base:(.text._ZN3tvm7runtime6vulkan12VulkanStream11SynchronizeEv[_ZN3tvm7runtime6vulkan12VulkanStream11SynchronizeEv]+0x39c): undefined reference to `vkWaitForFences'
> 
> /Users/user/Library/Android/sdk/ndk/21.1.6352462/toolchains/llvm/prebuilt/darwin-x86_64/sysroot/usr/include/c++/v1/__mutex_base:(.text._ZN3tvm7runtime6vulkan12VulkanStream11SynchronizeEv[_ZN3tvm7runtime6vulkan12VulkanStream11SynchronizeEv]+0x3a0): undefined reference to `vkWaitForFences'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::VulkanStream::Synchronize()':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:164: undefined reference to `vkResetCommandBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:164: undefined reference to `vkResetCommandBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:165: undefined reference to `vkResetFences'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:165: undefined reference to `vkResetFences'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:173: undefined reference to `vkBeginCommandBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:173: undefined reference to `vkBeginCommandBuffer'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `VulkanStream':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:55: undefined reference to `vkCreateCommandPool'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:55: undefined reference to `vkCreateCommandPool'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:63: undefined reference to `vkAllocateCommandBuffers'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:63: undefined reference to `vkAllocateCommandBuffers'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:70: undefined reference to `vkCreateFence'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:70: undefined reference to `vkCreateFence'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:77: undefined reference to `vkBeginCommandBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:77: undefined reference to `vkBeginCommandBuffer'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::VulkanModuleNode::GetPipeline(unsigned long, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, unsigned long)':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1143: undefined reference to `vkCreateShaderModule'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1143: undefined reference to `vkCreateShaderModule'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1217: undefined reference to `vkCreateDescriptorSetLayout'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1217: undefined reference to `vkCreateDescriptorSetLayout'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1229: undefined reference to `vkCreateDescriptorPool'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1229: undefined reference to `vkCreateDescriptorPool'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1238: undefined reference to `vkAllocateDescriptorSets'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1238: undefined reference to `vkAllocateDescriptorSets'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1262: undefined reference to `vkCreatePipelineLayout'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1262: undefined reference to `vkCreatePipelineLayout'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1279: undefined reference to `vkCreateComputePipelines'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1279: undefined reference to `vkCreateComputePipelines'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `~VulkanDeviceAPI':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:(.text._ZN3tvm7runtime6vulkan15VulkanDeviceAPID2Ev[_ZN3tvm7runtime6vulkan15VulkanDeviceAPID2Ev]+0x3c): undefined reference to `vkDestroyDevice'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:(.text._ZN3tvm7runtime6vulkan15VulkanDeviceAPID2Ev[_ZN3tvm7runtime6vulkan15VulkanDeviceAPID2Ev]+0x40): undefined reference to `vkDestroyDevice'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:239: undefined reference to `vkDestroyInstance'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:239: undefined reference to `vkDestroyInstance'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::VulkanDeviceAPI::FreeDataSpace(DLDevice, void*)':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:263: undefined reference to `vkDestroyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:263: undefined reference to `vkDestroyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:264: undefined reference to `vkFreeMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:264: undefined reference to `vkFreeMemory'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `tvm::runtime::vulkan::VulkanDeviceAPI::CopyDataFromTo(void const*, unsigned long, void*, unsigned long, unsigned long, DLDevice, DLDevice, DLDataType, void*)':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:351: undefined reference to `vkFlushMappedMemoryRanges'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:351: undefined reference to `vkFlushMappedMemoryRanges'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:333: undefined reference to `vkInvalidateMappedMemoryRanges'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:333: undefined reference to `vkInvalidateMappedMemoryRanges'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `operator()':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:295: undefined reference to `vkCmdCopyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:295: undefined reference to `vkCmdCopyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:305: undefined reference to `vkCmdPipelineBarrier'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:305: undefined reference to `vkCmdPipelineBarrier'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:322: undefined reference to `vkCmdCopyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:322: undefined reference to `vkCmdCopyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:363: undefined reference to `vkCmdPipelineBarrier'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:363: undefined reference to `vkCmdPipelineBarrier'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:371: undefined reference to `vkCmdCopyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:371: undefined reference to `vkCmdCopyBuffer'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `VulkanDescriptorTemplateKHRFunctions':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_common.h:105: undefined reference to `vkGetDeviceProcAddr'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_common.h:105: undefined reference to `vkGetDeviceProcAddr'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `VulkanGetBufferMemoryRequirements2Functions':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_common.h:124: undefined reference to `vkGetDeviceProcAddr'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_common.h:124: undefined reference to `vkGetDeviceProcAddr'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `~VulkanModuleNode':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1100: undefined reference to `vkDestroyPipeline'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1100: undefined reference to `vkDestroyPipelineLayout'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1100: undefined reference to `vkDestroyDescriptorPool'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1100: undefined reference to `vkDestroyPipeline'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1100: undefined reference to `vkDestroyPipelineLayout'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1100: undefined reference to `vkDestroyDescriptorPool'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1100: undefined reference to `vkDestroyDescriptorSetLayout'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1100: undefined reference to `vkDestroyDescriptorSetLayout'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:(.text._ZN3tvm7runtime6vulkan16VulkanModuleNodeD2Ev[_ZN3tvm7runtime6vulkan16VulkanModuleNodeD2Ev]+0x78): undefined reference to `vkDestroyShaderModule'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:(.text._ZN3tvm7runtime6vulkan16VulkanModuleNodeD2Ev[_ZN3tvm7runtime6vulkan16VulkanModuleNodeD2Ev]+0x7c): undefined reference to `vkDestroyShaderModule'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `~VulkanThreadEntry':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:(.text._ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev[_ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev]+0xc0): undefined reference to `vkDestroyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:(.text._ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev[_ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev]+0xc4): undefined reference to `vkFreeMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:(.text._ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev[_ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev]+0xc8): undefined reference to `vkUnmapMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:(.text._ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev[_ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev]+0xcc): undefined reference to `vkDestroyBuffer'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:(.text._ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev[_ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev]+0xd0): undefined reference to `vkFreeMemory'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:(.text._ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev[_ZN3tvm7runtime6vulkan17VulkanThreadEntryD2Ev]+0xd4): undefined reference to `vkUnmapMemory'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `~VulkanStream':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:81: undefined reference to `vkDestroyFence'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:81: undefined reference to `vkDestroyFence'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:82: undefined reference to `vkDestroyCommandPool'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan_stream.h:82: undefined reference to `vkDestroyCommandPool'
> 
> CMakeFiles/tvm_runtime_objs.dir/src/runtime/vulkan/vulkan.cc.o: In function `operator()':
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1455: undefined reference to `vkCmdBindPipeline'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1455: undefined reference to `vkCmdBindPipeline'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1465: undefined reference to `vkCmdPushConstants'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1465: undefined reference to `vkCmdPushConstants'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1470: undefined reference to `vkCmdDispatch'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1470: undefined reference to `vkCmdDispatch'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1477: undefined reference to `vkCmdPipelineBarrier'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1477: undefined reference to `vkCmdPipelineBarrier'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1507: undefined reference to `vkUpdateDescriptorSets'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1507: undefined reference to `vkUpdateDescriptorSets'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1512: undefined reference to `vkCmdBindPipeline'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1512: undefined reference to `vkCmdBindPipeline'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1513: undefined reference to `vkCmdBindDescriptorSets'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1513: undefined reference to `vkCmdBindDescriptorSets'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1521: undefined reference to `vkCmdPushConstants'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1521: undefined reference to `vkCmdPushConstants'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1526: undefined reference to `vkCmdDispatch'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1526: undefined reference to `vkCmdDispatch'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1533: undefined reference to `vkCmdPipelineBarrier'
> 
> /Users/user/Documents/AlgofaceWork/TVM/tvm/src/runtime/vulkan/vulkan.cc:1533: undefined reference to `vkCmdPipelineBarrier'
> 
> clang++: **error:** linker command failed with exit code 1 (use -v to see invocation)

Could you please give me additional comments where I made mistake

WildTaras · June 1, 2021, 8:50pm

@csullivan if you will show config.cmake for compiling libtvm_runtime.so for Android device, could you please elaborate experience with linking Vulkan library. Did you change anything, which env variables did you set etc. I will be grateful for any help.

WildTaras · June 2, 2021, 5:58am

Also, I find out one weird thing, when I use function

find_library(Vulkan_LIBRARY NAMES vulkan vulkan-1 PATHS ${__vulkan_sdk}/lib)

it always return me Vulkan_LIBRARY-NOTFOUND. It seems, that there are no library at this path, but I made

message(STATUS "Vulkan path for checking" ${__vulkan_sdk})

and after that check with ls utility path of __vulkan_sdk + /lib and got next output:

   cmake                       libLLVMPassPrinters.a
    libclang.a                  libLLVMProfileData.a
    libclangAnalysis.a          libLLVMScalarOpts.a
    libclangAST.a               libLLVMSupport.a
    libclangASTMatchers.a       libLLVMTableGen.a
    libclangBasic.a             libLLVMTarget.a
    libclangCodeGen.a           libLLVMTransformUtils.a
    libclangDriver.a            libLLVMVectorize.a
    libclangEdit.a              libMachineIndependent.a
    libclangFormat.a            libOGLCompiler.a
    libclangFrontend.a          libOSDependent.a
    libclangFrontendTool.a      libshaderc.a
    libclangIndex.a             libshaderc_combined.a
    libclangLex.a               libshaderc_shared.so
    libclangParse.a             libshaderc_shared.so.1
    libclangRewrite.a           libshaderc_util.a
    libclangRewriteFrontend.a   libSPIRV.a
    libclangSema.a              libspirv-cross-c.a
    libclangSPIRV.a             libspirv-cross-core.a
    libclangTooling.a           libspirv-cross-cpp.a
    libclangToolingCore.a       libspirv-cross-c-shared.so
    libdxclib.a                 libspirv-cross-c-shared.so.0
    libdxcompiler.so            libspirv-cross-c-shared.so.0.45.0
    libdxcompiler.so.3.7        libspirv-cross-glsl.a
    libGenericCodeGen.a         libspirv-cross-hlsl.a
    libglslang.a                libspirv-cross-msl.a
    libHLSL.a                   libspirv-cross-reflect.a
    libLLVMAnalysis.a           libspirv-cross-util.a
    libLLVMAsmParser.a          libSPIRV-Tools.a
    libLLVMBitReader.a          libSPIRV-Tools-link.a
    libLLVMBitWriter.a          libSPIRV-Tools-opt.a
    libLLVMCore.a               libSPIRV-Tools-reduce.a
    libLLVMDxcSupport.a         libSPIRV-Tools-shared.so
    libLLVMDXIL.a               libSPVRemapper.a
    libLLVMDxilContainer.a      libVkLayer_api_dump.so
    libLLVMDxilPIXPasses.a      libVkLayer_demo_layer.so
    libLLVMDxilRootSignature.a  libVkLayer_device_simulation.so
    libLLVMDxrFallback.a        libVkLayer_gfxreconstruct.so
    libLLVMHLSL.a               libVkLayer_khronos_synchronization2.so
    libLLVMInstCombine.a        libVkLayer_khronos_validation.so
    libLLVMipa.a                libVkLayer_monitor.so
    libLLVMipo.a                libVkLayer_screenshot.so
    libLLVMIRReader.a           libVkLayer_starter_layer.so
    libLLVMLinker.a             libVkLayer_utils.a
    libLLVMLTO.a                libvulkan.so
    libLLVMMiniz.a              libvulkan.so.1
    libLLVMMSSupport.a          libvulkan.so.1.2.170
    libLLVMOption.a             pkgconfig
    libLLVMPasses.a

As you can see, libvulkan.so is existed. The same problem with Vulkan_SPIRV_TOOLS_LIBRARY. It is the reason, why I set these paths manually. Maybe this trouble can give more information about the global problem?

P.S. This problem exist both on macOS and Ubuntu. Above log I got from Ubuntu.

masahi · June 2, 2021, 7:42am

Can you compile and run other normal vulkan applications? I have export VULKAN_SDK=/home/masa/1.2.162.1/x86_64 in my bashrc and this is I all need to get vulkan working with TVM on ubuntu.

WildTaras · June 2, 2021, 8:33am

@masahi You know, you are completely right. Looks please, if I use:

cmake -DUSE_LLVM=llvm-config-8 \
      -DUSE_RPC=ON \
      -DUSE_SORT=ON \
      -DUSE_VULKAN=ON \
      -DUSE_GRAPH_EXECUTOR=ON \
      ..
make -j10

everything are working fine. It finds Vulkan with all necessary includes and libraries. I have compiled tvm_runtime library successfully. But, for Android version of tvm_runtime, I need to use this command line:

cmake -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=armeabi-v7a -DANDROID_NATIVE_API_LEVEL=23 -DANDROID_ARM_NEON=ON ..

What I want to admit, when I use _DCMAKE_TOOLCHAIN_FILE with it’s value, It cause problems with Vulkan. I use this toolchain from android_ndk_r21e, because it was mentioned in this post:

gist.github.com

https://gist.github.com/apivovarov/c3d35e53103b25d625481d2bccf4f606

build-tvm-runtime-for-android-cmake.md

# Build TVM runtime for Android using cmake

#### Install cmake (for example 1.19.6)
https://cmake.org/download/

#### Download and extract NDK (e.g. r21e)
https://developer.android.com/ndk/downloads

#### Set env var NDK (chage the path to correct one)
```

This file has been truncated. show original

Could you please ask someone who has build tvm_runtime for Android, to give me a hint, which toolchain I need to use, or maybe it has another way to build?