I am a newer of tvm, i have read doc of deploy. but i still have confuse. after export graph,lib,params files. how to deploy them on target device. do i still install tvm like [ Install from Source] part of doc on tearget device? such as clone project, download llvm and build…then Python Package? Or there is a esay way to do it?
That’s not necessary, you can simply include tvm_runtime_pack.cc in your source files along with tvm header files. That’s all you need to deploy, for example, to
thx for your reply, i use Python to deloy, and Python need install tvm to
import tvm, i try to copy tvm python/tvm even cp topi,nnvm. and set env on target device(a ubuntu machine), is this right way? it cause some lib dependence errors, such as g++ needed…
It’s not advisable to copy tvm build from a device to another, always rebuild it from source if you actually need
So it still need rebuild on target device? that means some build tools should be installed on target device, for example g++ ,cmake, llvm etc. That doesn’t sound reasonable. and not lightly
You don’t need llvm support to run inference, you only need tvm runtime.
thx very much, what do you mean
don’t need llvm support to run inference .
i am new, forgive my stupid question. i build model with
target = ‘llvm’ target_host = ‘llvm’
That’s correct but you don’t build the model on your target device. You can build it on your main machine (that has llvm support and everything else you need at compilation time) and deploy the
.params files to your target device (that needs tvm runtime only). Imagine deploying a model to Android, you crosscompile the model on your main machine and then just run the compiled model on Android using tvm runtime.
Yes, i do the same thing like you said, i build on ubuntu 14, and deploy it on ubuntu 14 too, and load model with Python like
loaded_lib = tvm.module.load(path_lib), But it cause a err:
FileNotFoundError: [Errno 2] No such file or directory: ‘g++’, after i install the g++ on target ubuntu 14, it works? is g++ necessary?
I’m not familiar with deploying using Python, I’ve only used C++ and Java. Maybe someone else can help with your question.
I fix the issue, i read the code, becasue i export the lib to deploy_lib.tar, so when load the lib , tvm extract the lib.o file from .tar, and use g++ to complie to .so file. i should export lib to deploy_lib.so. it works for me.
Here is how I deploy graph, params and lib
thisdir = os.path.dirname(os.path.abspath(__file__)) path_lib = os.path.join(thisdir, "deploy.so") lib.export_library(path_lib) with open(os.path.join(thisdir, "deploy.json"), "w") as fo: fo.write(graph.json()) with open(os.path.join(thisdir, "deploy.params"), "wb") as fo: fo.write(tvm.relay.save_param_dict(params)) with open(os.path.join(thisdir, "cat.bin"), "w") as fo: fo.write(x.astype(np.float32).tobytes())
what is x? input of model?
It’s the input, you can ignore
x if you only want to save graph, lib and params.
I am trying to deploy the model with c++. How could I obtain the required headers please?
you can either:
- copy a libtvm.so for inference (you need obtain such libtvm.so on target device);
- just using https://github.com/apache/tvm/blob/main/apps/howto_deploy/tvm_runtime_pack.cc aalong with it’s related files to build for inference.