[DISCUSS] Combine multi SYSTEM-LIB module libs to one

Questions
#1

Deploy models on Android & iOS platform must use system-lib. Avoid usage of dlopen.
bundle_deploy demonstrated how to deploy a model which build target contains system-lib. Usually we deploy more than one model which tvm not support.

Working workaround: How to deploy two different tvm compiled model in c++ statically?
But this workaround has some limitations. AlterOpLayout pass must be disabled. AlterOpLayout pass can double the inference speed of some model.

Therefore I want to add Combine multi SYSTEM-LIB module libs to one function to tvm .
Work flow:

  1. Build each models separately
  2. Rename duplicate node name in all model’s GraphRuntimeFactory(json node name, params key, function name in lib), re-generate llvm module
  3. Generate modified graph json, params and a all in one lib
  4. Use runtime.SystemLib to load all in one lib
  5. Use tvm.graph_runtime.create & load_params to create each model’s runtime separately
2 Likes
#2

Any update about this topic?

#3
2 Likes
#6

Do we have something like what you proposed but for more recent version of TVM @fantasyRqg

#7

mlc llm now comes with multi system lib support that can serve as a good reference https://github.com/mlc-ai/mlc-llm

#8

thanks for your response @tqchen can you point to specific example where I can reference. With TVM, i’m currently able to combine 2 models into 1 systemlib with llvm but when I compile with cuda I got an error. Reference: Check failed: (f != nullptr) is false, Cannot find function when building 2 systemlibs with CUDA