# Status on quantization in TVM **URL:** \ **Category:** Questions\ **Created:** [December 9, 2021, 2:53pm UTC](https://discuss.tvm.apache.org/t/status-on-quantization-in-tvm/11668 "2021-12-09T14:53:13Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![JosseVanDelm](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/jossevandelm/32/2083_2.png) [@JosseVanDelm](https://discuss.tvm.apache.org/u/JosseVanDelm)\ **Post date:** [December 9, 2021, 2:53pm UTC](https://discuss.tvm.apache.org/t/status-on-quantization-in-tvm/11668/1 "2021-12-09T14:53:13Z")
Hi all, I’m working together with @wiebevr on using TVM for an embedded SoC with a Neural Network accelerator. Currently we’re looking into how we can best implement quantization to our tvm-fork. We’re looking into both pre-quantized models as well as quantization done in TVM whatever would be the easiest. However it’s quite hard to find information on the best way to get started with this. We already found some implementations scattered throughout this forum: > [@\[RFC\]\[Quantization\] A new quantization framework in TVM: initial RFC (1/4)](https://discuss.tvm.apache.org/t/rfc-quantization-a-new-quantization-framework-in-tvm-initial-rfc-1-4/9775/3): > > This looks like great work, thanks for the RFC! I agree that it’s very valuable for there to be a stage in the Relay lowering where the ‘QNN-ness’ is explicit - we’ve got both hardware and performance libraries which accelerate quantized operators specifically. One of the things that complicates our ability to match QNN operators though is the inconsistent way they’re represented. For instance, for QNN convolution we must match qnn.conv2d -\> bias\_add -\> qnn.requantize whereas for something lik… > > > This PR introduces a new framework for quantization to TVM. For details and expl…anation of the code structure, please see https://discuss.tvm.apache.org/t/rfc-quantization-quantization-in-tvm/9161. > > Please let me know if you have any questions or comments. > > Also, I'd like to thank @mbrookhart and @jwfromm for their mentorship on this project! > [@INT8 Quantization - Code generation for backends](https://discuss.tvm.apache.org/t/int8-quantization-code-generation-for-backends/518/4): > > Similar observation as in your other post - it might be good to think about supporting arbitrary precision operators at the graph level since there is work on supporting bit-serial convolutions on RPi, and also quantized operators in FPGAs, accelerators etc. The challenge with arbitrary precision is data layout, and specifically how to pack data into standard 8bit/32bit words. I’m happy to discuss some ideas, and present some concrete scenarios down the road as we continue our work on quantiza… > [@\[RFC\] Improvements to Automatic Quantization for Bare-Metal](https://discuss.tvm.apache.org/t/rfc-improvements-to-automatic-quantization-for-bare-metal/7108): > > For bare-metal devices, it is desirable (for both space and performance reasons) to have a network that consists entirely of integral data types (most often int8). However, the automatic integer quantization mechanism in Relay does not serve this use case for two reasons: Inputs are assumed to be float32, so they are quantized at the network’s prefix, and outputs are forced into float32, so they are dequantized at the network’s suffix. The quantization pass is geared towards only the most tim… [https://github.com/apache/tvm/tree/main/gallery/how\_to/deploy\_models](https://github.com/apache/tvm/tree/main/gallery/how_to/deploy_models) I also found the tutorials in the `gallery/how_to/deploy_models folder` (`deploy_quantized.py, deploy_prequantized.py, deploy_prequantized_tflite.py`), but I had some issues getting them to work, and was wondering if they were still being kept up to date. We’re also looking into int2 and int4 deployment, would the [BYOD tutorial](https://tvm.apache.org/docs/how_to/extend_tvm/bring_your_own_datatypes.html) be a good fit for this? This search has been a bit overwhelming, any help is greatly appreciated! Thank you in advance! Josse and Wiebe ---
**Author:** ![masahi](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/masahi/32/74_2.png) [@masahi](https://discuss.tvm.apache.org/u/masahi)\ **Post date:** [December 9, 2021, 10:40pm UTC](https://discuss.tvm.apache.org/t/status-on-quantization-in-tvm/11668/2 "2021-12-09T22:40:23Z")
> [@JosseVanDelm](#): > > I also found the tutorials in the `gallery/how_to/deploy_models folder` (`deploy_quantized.py, deploy_prequantized.py, deploy_prequantized_tflite.py`), but I had some issues getting them to work, and was wondering if they were still being kept up to date. Tutorials are run on CI, so if you use the same framework version used in CI it should work. I’m aware that `deploy_prequantized.py,` didn’t work with the recent PyTorch versions, but I’ve just merged the fix this week to make it work on PyTorch 1.10. > [@JosseVanDelm](#): > > We’re also looking into int2 and int4 deployment, would the [BYOD tutorial](https://tvm.apache.org/docs/how_to/extend_tvm/bring_your_own_datatypes.html) be a good fit for this? We are not ready for sub-byte e2e deployment. There some int4 CUDA kernels in topi, but they don’t work with Relay yet. For example, constant folding doesn’t work with sub-byte or bf16. > [@JosseVanDelm](#): > > This search has been a bit overwhelming, any help is greatly appreciated! I know, I was in the same situation when I started investigating the quantization situation in late 2019. We should have a good support for prequantized models in tflite or pytorch by now, and this year we began to add ONNX prequantized support. Quantized ONNX models can be created via the `tf2onnx` tool or using [the quantization tool in ONNXRuntime](https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/quantization). Quantization by TVM, however, is not in a good situation. First of all, we should acknowledge that quantization is a very hard problem on its own right. The one that’s already in TVM was implemented around 2018, the consensus seems to be that it doesn’t work beyond its demo (imagenet). As you found in the forum, earlier this year there was a proposal to introduce a new quantization system. But as far as I know, there has been no real development of that RFC. And often, quantization-aware training (QAT) is required for preserving sane accuracy for some int8 models (BERT, mobilenet, efficientnet type models etc) and most int4 models. Since we don’t support training even fp32 models yet, I don’t expect QAT by TVM would happen any time soon. My recommendation for quantization flow is to quantize models using DL frameworks, and rely on prequantized model support in TVM (actually, this is the only reasonable way). Both TF and PT have reasonable support for quantization (both PTQ and QAT). Several companies are using this flow in production (ARM, edgecortix). Outside of DL frameworks’ built-in quantization functionalities, there are also many third-party quantization tools that take TF/PT models, quantize it by PTQ or QAT, and output the quantized model in ONNX. I believe those tools are developed because framework built-in support for quantization may not be sufficient for many HW vendors. If you are serious about quantization, I encourage you to take a look at them. Since they output ONNX models, those tools are compatible with TVM. I’ve added some links below. > **[GitHub - quic/aimet: AIMET is a library that provides advanced...](https://github.com/quic/aimet)** > > AIMET is a library that provides advanced quantization and compression techniques for trained neural network models. > **[GitHub - openvinotoolkit/nncf: Neural Network Compression Framework for enhanced...](https://github.com/openvinotoolkit/nncf)** > > Neural Network Compression Framework for enhanced OpenVINO™ inference > **[TensorRT/tools/pytorch-quantization at main · NVIDIA/TensorRT](https://github.com/NVIDIA/TensorRT/tree/main/tools/pytorch-quantization)** > > NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT. - NVIDIA/TensorRT > **[GitHub - Xilinx/brevitas: Brevitas: neural network quantization in PyTorch](https://github.com/Xilinx/brevitas)** > > Brevitas: neural network quantization in PyTorch ---
**Author:** ![ramana-arm](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/r/ecb155/32.png) [@ramana-arm](https://discuss.tvm.apache.org/u/ramana-arm)\ **Post date:** [December 10, 2021, 6:07pm UTC](https://discuss.tvm.apache.org/t/status-on-quantization-in-tvm/11668/3 "2021-12-10T18:07:21Z")
The TFlite frontend of TVM isn’t yet fully featured to cover all the quantized behaviour in TFlite. Additionally there are open tickets on adding more quantized operator support to the Tflite frontend . See this tracking ticket. > > > As part of https://github.com/apache/tvm/pull/9165, we recently enabled support …for quantized operators in the TFLite frontend "unary element-wise" function, that does generic support for a few operators. > > That opens the door for more operators to be supported easily, just pending the addition of tests and logically \_unlocking\_ the conditional block that is keeping the quantised version of the operator to be unsupported. See the example as implemented in https://github.com/apache/tvm/pull/9165. > > This is a tracking issue for new unary element-wise operators to be supported: > > \- \[x\] \`GREATER\` #12754 @dchauhan-arm > \- \[x\] \`EQUAL\` #11520 @dchauhan-arm > \- \[x\] \`CEIL\` #10566 by @leeexyz > \- \[x\] \`ROUND\` #10566 by @leeexyz > \- \[x\] \`FLOOR\` #10566 by @leeexyz > \- \[x\] \`EXP\` #10566 by @leeexyz > \- \[x\] \`LOG\` #10566 by @leeexyz > \- \[x\] \`SIN\` #9478 by @onkar-sima-ai > \- \[x\] \`TAN\` #10566 by @leeexyz > \- \[x\] \`COS\` #10566 by @leeexyz > \- \[x\] \`SQRT\` #9258 by @elinx > \- \[x\] \`RSQRT\` #9165 by @ophirfrish > \- \[x\] \`NEG\` #9404 by @ophirfrish > > For each operator above, there are actions to be taken: > 1. Remove the conditional block \`if self.is\_quantized(op): ...\` > 2. Implement unit tests > 3. ask any \[committer\](https://github.com/apache/tvm/blob/main/CONTRIBUTORS.md) in the project to update this issue, reflecting the PR to support the operators on the list above > > \#9165 can be used as an example. > > Resources: > \* TFlite frontend: https://github.com/apache/tvm/blob/main/python/tvm/relay/frontend/tflite.py Ramana