Hello,
I am using TVM to integrate a machine learning model in my code. I already tuned my model on my CPU using AutoTVM, which I integrated in my C++ code following the cpp deploy tutorial.
The problem is that I am not having good performances. I linked my model as a dynamic library using the tvm::runtime::LoadFromFile function. My perfs are about x2.5 longer than using cppflow, the TensorFlow C API to run models. In cpp flow, the batch size (or input size) is dynamically set, whereas I compiled my models with static batch sizes, and I still get worse performance than with cppflow.
I have a few guesses :
- I am running on a Intel Xeon Platnium 8620L server CPU that has a lot of RAM (3Tb), I didn’t find any Intel CPU benchmarks. But I did for AMD, ARM, and NVIDIA GPUs. Maybe I should be trying on AMD CPUs ?
- My input sizes are 60, 200, 3000 and 3536, which is too small to fully use my hardware’s capabilities. But my guess would be that the smaller the faster, if it can run a batch size of 2^20, it should be able to run fast a batch size of 3000 ??
- I am not sure what is the difference between the LoadFromFile of a dynamic lib and the (*tvm::runtime::Registry::Get(“runtime.SystemLib”))(); method with a system library. But anyways, I am not timing this but only the “run” function tvm::runtime::PackedFunc run = gmod.GetFunction(“run”);, so it shouldn’t be a problem, but maybe TVM internally does some lazy programming and it loads in memory the model when I call the “run” function ?
- I tried to rebuild the TVM runtime lib without logging library, think that maybe this was the problem, but nothing changed
These guesses are only intuition, to give you an idea of where I am stuck. If anyone has any idea on where I should be looking, I would be great help ! I really feel like I just forgot to do a simple thing to get TVM fully working (maybe a compilation flag?, a fucntion somewhere ?), because everything else works fine (AutoTVM, the integration, the NDArrays structures, etc.).
Please do send me a message if you want to discuss in private about this.
EDIT : I am going to recompile TVM using the MKL library, I’ll let you know if it worked