How to deploy tvm(libtvm_runtime) with saved graph, param, lib?

Questions
#1

I am a newer of tvm, i have read doc of deploy. but i still have confuse. after export graph,lib,params files. how to deploy them on target device. do i still install tvm like [ Install from Source] part of doc on tearget device? such as clone project, download llvm and build…then Python Package? Or there is a esay way to do it?

1 Like
#2

That’s not necessary, you can simply include tvm_runtime_pack.cc in your source files along with tvm header files. That’s all you need to deploy, for example, to x86 target.

#3

thx for your reply, i use Python to deloy, and Python need install tvm to import tvm, i try to copy tvm python/tvm even cp topi,nnvm. and set env on target device(a ubuntu machine), is this right way? it cause some lib dependence errors, such as g++ needed…

#4

It’s not advisable to copy tvm build from a device to another, always rebuild it from source if you actually need libtvm.so

#5

So it still need rebuild on target device? that means some build tools should be installed on target device, for example g++ ,cmake, llvm etc. That doesn’t sound reasonable. and not lightly

#6

You don’t need llvm support to run inference, you only need tvm runtime.

1 Like
#7

thx very much, what do you mean don’t need llvm support to run inference . i am new, forgive my stupid question. i build model with

target = ‘llvm’ target_host = ‘llvm’

#8

That’s correct but you don’t build the model on your target device. You can build it on your main machine (that has llvm support and everything else you need at compilation time) and deploy the .so, .json and .params files to your target device (that needs tvm runtime only). Imagine deploying a model to Android, you crosscompile the model on your main machine and then just run the compiled model on Android using tvm runtime.

#9

Yes, i do the same thing like you said, i build on ubuntu 14, and deploy it on ubuntu 14 too, and load model with Python like loaded_lib = tvm.module.load(path_lib), But it cause a err: FileNotFoundError: [Errno 2] No such file or directory: ‘g++’, after i install the g++ on target ubuntu 14, it works? is g++ necessary?

#10

I’m not familiar with deploying using Python, I’ve only used C++ and Java. Maybe someone else can help with your question.

#11

I fix the issue, i read the code, becasue i export the lib to deploy_lib.tar, so when load the lib , tvm extract the lib.o file from .tar, and use g++ to complie to .so file. i should export lib to deploy_lib.so. it works for me.

1 Like
#12

Here is how I deploy graph, params and lib

thisdir = os.path.dirname(os.path.abspath(__file__))
path_lib = os.path.join(thisdir, "deploy.so")
lib.export_library(path_lib)
with open(os.path.join(thisdir, "deploy.json"), "w") as fo:
    fo.write(graph.json())
with open(os.path.join(thisdir, "deploy.params"), "wb") as fo:
    fo.write(tvm.relay.save_param_dict(params))
with open(os.path.join(thisdir, "cat.bin"), "w") as fo:
    fo.write(x.astype(np.float32).tobytes())
1 Like
#13

what is x? input of model?

#14

It’s the input, you can ignore x if you only want to save graph, lib and params.

#15

Hi,

I am trying to deploy the model with c++. How could I obtain the required headers please?

#16

you can either: