# Can TVM now support batched inference? Autotvm runs twice as long as tensorflow **URL:** \ **Category:** Questions\ **Created:** [April 17, 2020, 2:19am UTC](https://discuss.tvm.apache.org/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405 "2020-04-17T02:19:27Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![adobay](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/a/51bf81/32.png) [@adobay](https://discuss.tvm.apache.org/u/adobay)\ **Post date:** [April 17, 2020, 2:19am UTC](https://discuss.tvm.apache.org/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/1 "2020-04-17T02:19:27Z")
I have a tensorflow model. The cpu inference performance is poor when the batch is 500 online. After using autoTVM optimization, the performance of 500 times is much worse than tensorflow 500 batch. Can TVM support batch inference? With 50 times 1000 batch, tensorflow cost 8.62s on my mac, while autotvm cost 15.85s. ---
**Author:** ![comaniac](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/comaniac/32/899_2.png) [@comaniac](https://discuss.tvm.apache.org/u/comaniac)\ **Post date:** [April 17, 2020, 5:50am UTC](https://discuss.tvm.apache.org/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/2 "2020-04-17T05:50:47Z")
You can try to use batch 1 for tuning and 500 for inference. The time should be just around (batch size) \* (single batch inference time). Current TVM HCHW/NHWC conv2d does not tune the batch size, but some work is ongoing. ---
**Author:** ![adobay](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/a/51bf81/32.png) [@adobay](https://discuss.tvm.apache.org/u/adobay)\ **Post date:** [April 17, 2020, 9:59am UTC](https://discuss.tvm.apache.org/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/3 "2020-04-17T09:59:45Z")
My model does not contain conv2d, the most time-consuming op is nn.dense. Do you mean using optimized history to build the relay using batch 500 and then do inference? ---
**Author:** ![comaniac](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/comaniac/32/899_2.png) [@comaniac](https://discuss.tvm.apache.org/u/comaniac)\ **Post date:** [April 17, 2020, 3:30pm UTC](https://discuss.tvm.apache.org/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/4 "2020-04-17T15:30:21Z")
Dense is another issue tho. In this case you have to tune the model with batch size 500. Did you try graph tuner after tuning each op? Another option is enabling cBLAS for dense ops by setting `target=llvm -lib=cblas` ---
**Author:** ![adobay](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/a/51bf81/32.png) [@adobay](https://discuss.tvm.apache.org/u/adobay)\ **Post date:** [April 17, 2020, 3:48pm UTC](https://discuss.tvm.apache.org/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/5 "2020-04-17T15:48:01Z")
Thank you very much. Tonight I will try what you said. The graph tuner throwed an exception, so i only tuned each op… ---
**Author:** ![adobay](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/a/51bf81/32.png) [@adobay](https://discuss.tvm.apache.org/u/adobay)\ **Post date:** [April 20, 2020, 2:30am UTC](https://discuss.tvm.apache.org/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/6 "2020-04-20T02:30:22Z")
Thanks @comaniac, with batch size 500 and `llvm -mcpu=haswell -libs=cblas`, compared with tensorflow, tvm gets 2~3X performance improvement. But the graph tuner will still throw an exception. [https://github.com/apache/incubator-tvm/issues/5369](https://github.com/apache/incubator-tvm/issues/5369) ---
**Author:** ![comaniac](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/comaniac/32/899_2.png) [@comaniac](https://discuss.tvm.apache.org/u/comaniac)\ **Post date:** [April 20, 2020, 5:07pm UTC](https://discuss.tvm.apache.org/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/7 "2020-04-20T17:07:43Z")
I am not sure if graph tuner is still applicable when cBLAS is used. Maybe @kevinthesun could provide more details about it. ---
**Author:** ![kevinthesun](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/kevinthesun/32/11_2.png) [@kevinthesun](https://discuss.tvm.apache.org/u/kevinthesun)\ **Post date:** [April 20, 2020, 5:44pm UTC](https://discuss.tvm.apache.org/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/8 "2020-04-20T17:44:22Z")
You don’t need graph tuning while using cblas. ---
**Author:** ![adobay](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/a/51bf81/32.png) [@adobay](https://discuss.tvm.apache.org/u/adobay)\ **Post date:** [April 21, 2020, 1:58am UTC](https://discuss.tvm.apache.org/t/can-tvm-now-support-batched-inference-autotvm-runs-twice-as-long-as-tensorflow/6405/9 "2020-04-21T01:58:16Z")
Thanks, @kevinthesun, @comaniac