# \[RFC\] Building a new reproducible benchmark for TVM **URL:** \ **Category:** RFC\ **Created:** [November 21, 2020, 2:32am UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496 "2020-11-21T02:32:42Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![merrymercy](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/merrymercy/32/2150_2.png) [@merrymercy](https://discuss.tvm.apache.org/u/merrymercy)\ **Post date:** [November 21, 2020, 2:32am UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/1 "2020-11-21T02:32:42Z")
## Motivation Currently, TVM lacks an up-to-date and reproducible benchmark. The only benchmark is hosted at [tvm/apps/benchmark](https://github.com/apache/incubator-tvm/tree/main/apps/benchmark). However, this benchmark is too old and has several flaws. 1. The results were obtained 2 years ago. 2. The deep learning models are old. It does not include new models (e.g., BERT, EfficientNet) 3. The input format is TVM’s internal relay format. It does not use formats from high-level frameworks (e.g., pytorch, mxnet) or open exchange format (e.g., ONNX). 4. It does not cover Intel CPUs. 5. It only provides pre-tuned configurations by [tophub](https://github.com/tlc-pack/tophub), but does not provide the scripts to generate these configurations. This RFC aims at building a new open, reproducible bechmark for TVM. When the new benchmark is ready, we can run evaluation nightly and run auto-tuning weekly or monthly. ## Approach As the first step, we target three models, three hardware platforms and four code generation strategies. To make the comparision with other frameworks easier, we choose ONNX as the input model format. - models: resnet-50, mobilenet v2 and BERT with batch size 1 - hardware platforms: NVIDIA GPU, Intel CPU, ARM CPU - code generation strategies: autotvm, auto-scheduler, tvm + manual library, ONNX-runtime. All logs generated during the auto-tuning should be uploaded for future references. I created one a repo [TLCbench](https://github.com/tlc-pack/tlcbench) and opened a [roadmap](https://github.com/tlc-pack/tlcbench/issues/1#roadmap). I am seeking for contributors who are interested in helping me. ---
**Author:** ![ziheng](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/ziheng/32/188_2.png) [@ziheng](https://discuss.tvm.apache.org/u/ziheng)\ **Post date:** [November 21, 2020, 2:54am UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/2 "2020-11-21T02:54:14Z")
Great suggestion! Can we make it as a nightly/weekly regression test utils and also consider adding accuracy evaluation for quantization model into this loop? ---
**Author:** ![kevinthesun](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/kevinthesun/32/11_2.png) [@kevinthesun](https://discuss.tvm.apache.org/u/kevinthesun)\ **Post date:** [November 21, 2020, 5:02am UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/3 "2020-11-21T05:02:05Z")
Yeah. A performance regression test would be very nice. There are a lot of times we need to do binary search to find the commit causing regression. ---
**Author:** ![comaniac](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/comaniac/32/899_2.png) [@comaniac](https://discuss.tvm.apache.org/u/comaniac)\ **Post date:** [November 22, 2020, 1:18am UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/4 "2020-11-22T01:18:48Z")
Glad to see this is being planned! I could help on this as much as I can. One question/suggestion is that if we are going to have such formal benchmarking approach, maybe we can make it MLPref friendly so that everyone can use this TVM utility to run these models on the target platform and submit the results to MLPref. ---
**Author:** ![tqchen](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/tqchen/32/5_2.png) [@tqchen](https://discuss.tvm.apache.org/u/tqchen)\ **Post date:** [November 22, 2020, 2:33am UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/5 "2020-11-22T02:33:30Z")
It would also be great to consider output [https://tvm.apache.org/docs/dev/benchmark.html](https://tvm.apache.org/docs/dev/benchmark.html) and iterate on a common log format ---
**Author:** ![zhiics](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/zhiics/32/146_2.png) [@zhiics](https://discuss.tvm.apache.org/u/zhiics)\ **Post date:** [November 22, 2020, 3:18am UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/6 "2020-11-22T03:18:49Z")
It is really nice to add the regression tests against a selected set of models, since the down streams users usually have to spend quite amount of time to find the root cause once there is a regression. Or they have to sync the upstream codebase as frequent as possible and test regression locally. cc @jroesch, you may have some comments about the output format or the UX of the test infra. ---
**Author:** ![FrozenGene](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/frozengene/32/1114_2.png) [@FrozenGene](https://discuss.tvm.apache.org/u/FrozenGene)\ **Post date:** [November 22, 2020, 7:09am UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/7 "2020-11-22T07:09:32Z")
One question for the performance regression, how to judge the normal fluctuation, especially CPU? Like resnet50 maybe 20.00ms, but becomes 20.88ms after one pr? ---
**Author:** ![giuseros](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/giuseros/32/803_2.png) [@giuseros](https://discuss.tvm.apache.org/u/giuseros)\ **Post date:** [November 27, 2020, 6:36pm UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/8 "2020-11-27T18:36:52Z")
Hi @merrymercy, I think that this is a great idea! Few questions: - Do you think that packaging is a requirement for this work? The point is that if we don’t have binary packages available (where users can simply do `pip install tvm`) it would be hard for external users to reproduce the benchmark results - Are you planning to use `tvmc` under the hood? I believe that using `tvmc` would largely simplify scripts like `benchmark_autotvm.py`, etc… - Why adding a separate repository? Why not adding directly into the main one? - Why focusing on ONNX? I believe that ONNX support in TVM lacks quantization (correct me if I am wrong), which would be nice to test as well. Thanks, ---
**Author:** ![merrymercy](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/merrymercy/32/2150_2.png) [@merrymercy](https://discuss.tvm.apache.org/u/merrymercy)\ **Post date:** [January 12, 2021, 6:39pm UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/9 "2021-01-12T18:39:14Z")
Hi @giuseros, Thanks for the good suggestions. > [@giuseros](#): > > Do you think that packaging is a requirement for this work? The point is that if we don’t have binary packages available (where users can simply do `pip install tvm` ) it would be hard for external users to reproduce the benchmark results This is already addressed by [TLC pack](https://tlcpack.ai/), by which you can install tvm by pip > [@giuseros](#): > > Are you planning to use `tvmc` under the hood? I believe that using `tvmc` would largely simplify scripts like `benchmark_autotvm.py` , etc… No, I think script is more flexible. But I will investigate tvmc later. > [@giuseros](#): > > Why adding a separate repository? Why not adding directly into the main one? We can move it to mainline and replace the old one when it is mature. But now we keep it as a separate repo for faster development. > [@giuseros](#): > > Why focusing on ONNX? I believe that ONNX support in TVM lacks quantization (correct me if I am wrong), which would be nice to test as well. What other formats do you think are better? ---
**Author:** ![merrymercy](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/merrymercy/32/2150_2.png) [@merrymercy](https://discuss.tvm.apache.org/u/merrymercy)\ **Post date:** [January 12, 2021, 6:41pm UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/10 "2021-01-12T18:41:31Z")
I updated some initial results on [https://github.com/tlc-pack/TLCBench](https://github.com/tlc-pack/TLCBench). They include the performance of autoscheduler and autotvm on AWS c5.9xlarge and g4.4dnxlarge. Contributions are welcome for more model and backend coverage.