I would like to use TVM to develop a GPU pytorch module. However I cannot figure out how to best distribute the code. What is the easiest way to ensure that end-users can run the function? Do they need to install TVM from source? Is there a pip package for the runtime alone?
Unfortunately we don’t have any pip package at the moment. But a runtime only package sounds reasonable. cc @tqchen
I’d imagine you’d build TVM code outside of Torch first, and export a build artifact as shared lib. And from Torch you can load the TVM-generated shared lib in either python code or C++ extension. We don’t generate headers, users of a TVM-generated shared lib dynamically load it in their app at runtime.
Users don’t need to install TVM compiler component or LLVM, but they need TVM runtime. If using from python, users need to have TVM python module and libruntime.so (built from source or distributed with your package) locally, so that import tvm works. If using from Torch C++ extension, I think you can link against libruntime.so when you build your Torch extension.
I think this use case is interesting, but one thing I’m not clear is whether or not input shapes from Torch are expected to be fixed. We usually expect the input shapes to TVM to be fixed, since TVM can specialize the generated code accordingly.
Please correct me or elaborate @tqchen @yzhliu @jwfromm @kevinthesun @haichen
We are in the process of doing quite a bit of refactoring in terms of the runtime in this release cycle. We do hope to get some packaging story for 0.7 through pip or conda
Thanks. I wasn’t sure if I was missing something. It seems pretty heavyweight to assume the users have full TVM installed and the only other option seems to ship the generated CUDA code which seems messy.
Thanks for the awesome library, really interesting to use.
You are absolutely right. libtvmruntime only library would be a good way to go. Right now(before 0.7) we can do that from source by typing make runtime instead of make
TVM kernel supports dynamic shape, while rank of the shape has to be fixed. We did some experimental work before, to exhaust all the combination of shape rank + op attribute ahead of time and compile to .so. It’s doable but has some restriction.
1 Like