# Importing quantized PyTorch model and problems with quint and qint datatypes

**URL:** <https://discuss.tvm.apache.org/t/importing-quantized-pytorch-model-and-problems-with-quint-and-qint-datatypes/15029>\
**Category:** Questions\
**Created:** [May 31, 2023, 4:01pm UTC](https://discuss.tvm.apache.org/t/importing-quantized-pytorch-model-and-problems-with-quint-and-qint-datatypes/15029 "2023-05-31T16:01:19Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Necrotos](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/n/f9ae1b/32.png) [@Necrotos](https://discuss.tvm.apache.org/u/Necrotos)\
**Post date:** [May 31, 2023, 4:01pm UTC](https://discuss.tvm.apache.org/t/importing-quantized-pytorch-model-and-problems-with-quint-and-qint-datatypes/15029/1 "2023-05-31T16:01:19Z")

</div>

Hello,

I have a model that I quantized in eager mode in PyTorch. I have an implementation that doesn’t use the QuantStub and QuantDestub layers, as I want my model to directly accept inputs of dtype int8/uint8 without the quantization steps. The problem I am facing is that the model now expects inputs of type qint8 or quint8, but that causes issues when importing. Similar problem with the output: dtype is qint8 or quint8, but TVM can’t work with them. Is there a way to deal with this?

---

<div class="post-metadata">

**Author:** ![AQSingh](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/aqsingh/32/4666_2.png) [@AQSingh](https://discuss.tvm.apache.org/u/AQSingh)\
**Post date:** [May 31, 2023, 8:46pm UTC](https://discuss.tvm.apache.org/t/importing-quantized-pytorch-model-and-problems-with-quint-and-qint-datatypes/15029/2 "2023-05-31T20:46:45Z")

</div>

You may have to be more specific. Where is the error occurring?

I have been dealing with an issue that sounds similar. The short answer is that TVM doesn’t support quint8 input (errors out in convert\_data\_type call in PyTorch frontend of TVM’s \_get\_relay\_input\_vars). You can either change your PyTorch model definition to use float input/output or modify the relevant function in the frontend to “unroll” quint8 inputs to an uint8 tensor with a zero point and scale attached. This introduces a difference in model definitions from PyTorch to Relay (Relay representation of your graph would have uint8 input/output).

---

<div class="post-metadata">

**Author:** ![Necrotos](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/n/f9ae1b/32.png) [@Necrotos](https://discuss.tvm.apache.org/u/Necrotos)\
**Post date:** [May 31, 2023, 9:02pm UTC](https://discuss.tvm.apache.org/t/importing-quantized-pytorch-model-and-problems-with-quint-and-qint-datatypes/15029/3 "2023-05-31T21:02:44Z")

</div>

It is indeed the same error! It also occurs in the \_get\_relay\_input\_vars functions.

Can you elaborate what you mean with “unrolling quint8”? I am targeting an embedded device, so I would prefer to get rid of float numbers, if that is at all possible. I know that PyTorch has the `quantized_tensor.int_repr()` function, but I think that also cause some issues when importing the model.

---

<div class="post-metadata">

**Author:** ![masahi](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/masahi/32/74_2.png) [@masahi](https://discuss.tvm.apache.org/u/masahi)\
**Post date:** [June 1, 2023, 12:39am UTC](https://discuss.tvm.apache.org/t/importing-quantized-pytorch-model-and-problems-with-quint-and-qint-datatypes/15029/4 "2023-06-01T00:39:57Z")

</div>

`quint8` is an opaque PyTorch type, we cannot ingest such inputs.
