I have a tensorflow model. The cpu inference performance is poor when the batch is 500 online. After using autoTVM optimization, the performance of 500 times is much worse than tensorflow 500 batch. Can TVM support batch inference?
With 50 times 1000 batch, tensorflow cost 8.62s on my mac, while autotvm cost 15.85s.