# \[RFC\] Building a new reproducible benchmark for TVM

**URL:** <https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496>\
**Category:** RFC\
**Created:** [November 21, 2020, 2:32am UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496 "2020-11-21T02:32:42Z")\
**Posts on this page:** 1\
**Showing post:** 1

<div class="post-metadata">

**Author:** ![merrymercy](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/merrymercy/32/2150_2.png) [@merrymercy](https://discuss.tvm.apache.org/u/merrymercy)\
**Post date:** [November 21, 2020, 2:32am UTC](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496/1 "2020-11-21T02:32:42Z")

</div>

## Motivation

Currently, TVM lacks an up-to-date and reproducible benchmark. The only benchmark is hosted at [tvm/apps/benchmark](https://github.com/apache/incubator-tvm/tree/main/apps/benchmark). However, this benchmark is too old and has several flaws.

1. The results were obtained 2 years ago.
2. The deep learning models are old. It does not include new models (e.g., BERT, EfficientNet)
3. The input format is TVM’s internal relay format. It does not use formats from high-level frameworks (e.g., pytorch, mxnet) or open exchange format (e.g., ONNX).
4. It does not cover Intel CPUs.
5. It only provides pre-tuned configurations by [tophub](https://github.com/tlc-pack/tophub), but does not provide the scripts to generate these configurations.

This RFC aims at building a new open, reproducible bechmark for TVM. When the new benchmark is ready, we can run evaluation nightly and run auto-tuning weekly or monthly.

## Approach

As the first step, we target three models, three hardware platforms and four code generation strategies. To make the comparision with other frameworks easier, we choose ONNX as the input model format.

- models: resnet-50, mobilenet v2 and BERT with batch size 1
- hardware platforms: NVIDIA GPU, Intel CPU, ARM CPU
- code generation strategies: autotvm, auto-scheduler, tvm + manual library, ONNX-runtime.

All logs generated during the auto-tuning should be uploaded for future references.

I created one a repo [TLCbench](https://github.com/tlc-pack/tlcbench) and opened a [roadmap](https://github.com/tlc-pack/tlcbench/issues/1#roadmap). I am seeking for contributors who are interested in helping me.

---

_[View the full topic](https://discuss.tvm.apache.org/t/rfc-building-a-new-reproducible-benchmark-for-tvm/8496)._