# TF Lite quantized conv2d operator conversion **URL:** \ **Category:** Questions\ **Created:** [May 21, 2019, 12:00am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651 "2019-05-21T00:00:08Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![janimesh](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/j/b5e925/32.png) [@janimesh](https://discuss.tvm.apache.org/u/janimesh)\ **Post date:** [May 21, 2019, 12:00am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/1 "2019-05-21T00:00:08Z")
I am trying to understand how a pre-quantized TF Lite network can be converted to Relay. So, this is a Reference implementation of a TF Lite conv2d operator - [https://github.com/tensorflow/tensorflow/blob/92f84571edb98bdb31f1e7b1e23f17bf2c1438e9/tensorflow/lite/kernels/internal/reference/conv.h#L101](https://github.com/tensorflow/tensorflow/blob/92f84571edb98bdb31f1e7b1e23f17bf2c1438e9/tensorflow/lite/kernels/internal/reference/conv.h#L101) The key point is that there are zero\_point and scale for each input. This means that we cannot naively convert the conv2d of TF lite to conv2d of Relay. So, do we plan to convert a TF lite conv2d into four separate operators with quantize and dequantize before and after? @FrozenGene ---
**Author:** ![FrozenGene](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/frozengene/32/1114_2.png) [@FrozenGene](https://discuss.tvm.apache.org/u/FrozenGene)\ **Post date:** [May 21, 2019, 2:53am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/2 "2019-05-21T02:53:15Z")
In our implementation, we will have one op named `q_conv2d`, which will have more parameters compared with `conv2d`. The detail is: ```auto int input_zero_point; double input_scale; int kernel_zero_point; double kernel_scale; int output_zero_point; double output_scale; int output_min; int output_max; ``` We will not need `dequantize` operator. In TFLite’s quantization model, we will not see `dequantize` operator. However, for `q_conv2d`, when to complete int8 \* int8 -\> int32 output, we will `requantize` int32 to int8, i.e. the parameter input\_scale / kernel\_scale / output\_zero\_point / output\_scale / output\_min / output\_max does. We will have one op `q_conv2d_requantize` to handle it. ---
**Author:** ![janimesh](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/j/b5e925/32.png) [@janimesh](https://discuss.tvm.apache.org/u/janimesh)\ **Post date:** [May 21, 2019, 5:27am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/3 "2019-05-21T05:27:05Z")
Thanks @FrozenGene. I have two followup questions - Whats the current plan of expressing TVM compute for **q\_conv2d**? Do you plan to write it from scratch? Or can we break it down into simpler operators using a Relay pass like simplify\_inference (somewhat on the lines of gemmlowp). - Also thinking of generalizing this with other frameworks as well. Maybe, we should have a look at other frameworks and come up with one q\_conv2d that works for all. For TF lite, we don’t need dequantization, but for others we might. In that case, we might need to enhance/break q\_conv2d if it is not the right abstraction. ---
**Author:** ![FrozenGene](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/frozengene/32/1114_2.png) [@FrozenGene](https://discuss.tvm.apache.org/u/FrozenGene)\ **Post date:** [May 21, 2019, 5:53am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/4 "2019-05-21T05:53:33Z")
`q_conv2d`'s compute is very similar with `conv2d`. The changed place is we should care convolution computation part. i.e. data \* weight will be changed into (data - input\_zero\_point).astype("int32) \* (weight - kernel\_zero\_point).astype("int32). Then we call `q_conv2d_requantize` to quantize int32 into int8. Another place we should care is the pad. The pad value should be input\_zero\_point, not 0. I think we use one new op `q_conv2d` is reasonable. Because we found the optimization of `q_conv2d` has different places compared with `conv2d`. Yes. we also consider other frameworks even other libraries such as QNNPACK. When to design the API, I keep in mind the API could be compatible with QNNPACK , though I don’t try integrating with QNNPACK, but it should be. Come back to `dequantization`, In TFLite, we have one op named as `dequantize`, which does the dequantization work. So, if we need dequantization, we need implement `dequantize`. If other frameworks need `dequantize`, we could combine `q_conv2d + dequantize`. ---
**Author:** ![janimesh](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/j/b5e925/32.png) [@janimesh](https://discuss.tvm.apache.org/u/janimesh)\ **Post date:** [May 21, 2019, 6:06am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/5 "2019-05-21T06:06:16Z")
Do you think if we convert every multiply of `data * weight` to `data - input_zero_point).astype("int32) * (weight - kernel_zero_point).astype("int32)`, it would result in bad performance? Because we have to subtract each element and that gets in the way of using Fused-Multiply-Accmulate? For example, it seems very unlikely that I can use Intel VNNI instruction with this type of compute. That being said, I was able to work through the maths and break down the computation into actual conv2d between 2 int8 matrices and 3 other operators, which seems to me might be faster or more friendly to map to HW. I will write up the equations tomorrow in a pretty manner so that we can look at it together. We can have more discussion then. > Because we found the optimization of `q_conv2d` has different places compared with `conv2d` Will it be possible to shed more light on this line? Which optimizations are these? Relay or TVM schedule? In case of TVM schedule, are you talking about any particular target? Makes sense about the QNNPack and dequantization comment. Thanks for reminding about QNNPACK, will look into that as well. Other question - Do you care about cases, where the output\_min/max/scale etc are not given (does not happen in TF Lite)? In that case, we might need to calculate min/max with input/weight min/max. I don’t know if we should care about this scenario, but seems like MxNet has this configuration as well. ---
**Author:** ![FrozenGene](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/frozengene/32/1114_2.png) [@FrozenGene](https://discuss.tvm.apache.org/u/FrozenGene)\ **Post date:** [May 21, 2019, 6:33am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/6 "2019-05-21T06:33:21Z")
We previously dig deeply when we want to improve performance, but we find we can’t omit this. We use tensorize to improve the performance. i.e. Firstly LOAD 8 elements then subtract it, then compute, then requantize. QNNPACK does it like this way too. Our main focus target is ARM CPU. On ARM CPU, when to handle `q_conv2d`, if we want to get better performance compared with `spatial pack`, we have to tensorize. The optimization way is said before, we should LOAD 8 firstly, then subtract it, then compute, then requantize into int8. The `spatial_pack`'s llvm codegen is not this, it will be LOAD, SUB, LOAD SUB and so on. If you want to improve performance on Intel CPU, I think you should tensorize too. Output min / max / scale will always be given in TFLite model. If MXNet doesn’t have, we should calculate before pass into `q_conv2d`. ---
**Author:** ![FrozenGene](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/frozengene/32/1114_2.png) [@FrozenGene](https://discuss.tvm.apache.org/u/FrozenGene)\ **Post date:** [May 21, 2019, 6:41am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/7 "2019-05-21T06:41:50Z")
However, GEMMLOWP has one way we could control the `input_zero_point` / `kernel_zero_point`, i.e. we could just `data * weight -> convolution`. then compute the offset after GEMM. see: [https://github.com/google/gemmlowp/blob/master/doc/low-precision.md#efficient-handling-of-offsets](https://github.com/google/gemmlowp/blob/master/doc/low-precision.md#efficient-handling-of-offsets) But I haven’t analyzed it. But I think when we have `q_conv2d`, it could be done very easily. i.e. write new schedule / tensorize for specific target for `q_conv2d`. ---
**Author:** ![janimesh](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/j/b5e925/32.png) [@janimesh](https://discuss.tvm.apache.org/u/janimesh)\ **Post date:** [May 21, 2019, 5:41pm UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/8 "2019-05-21T17:41:12Z")
Thanks for the detailed explanation @FrozenGene. This is really helpful. Let me lay down both the options so that other can also comment.@eqy @ziheng @tqchen please let us know if you have any suggestions on this one. Please tag others who might be interested in this. Also, let me know if I should convert the type to RFC. # Background Suppose there are two matrices A and W, and we want to perform convolution operation on them and get output C. If A and W are both FP32 matrices, the calculation is > C(n, k, h, w) = Σc,r,s W(k, c, r, s) × A(n, c, h + r, w + s) where `n = batch_size, k = output channels, h = output height, w = output width` and `c = input_channels, r = kernel_height, s = kernel_width` Now, a quantized representation looks something like this > A = scale\_a × (QA - zp\_a) where A is FP32 tensor, Qa is (u)int8 tensor and FP32 scale\_a and int32 zp\_a (zero\_point) are quantization parameters for that tensor. Please refer to [this](https://github.com/google/gemmlowp/blob/master/doc/quantization.md) link to understand more about the quantization parameters. So, the above conv computation becomes > C(n, k, h, w) = Σc,r,s [scale\_w × (QW(k, c, r, s) - zp\_w)] × [scale\_a × (QA(n, c, h + r, w + s) - zp\_a)] > C(n, k, h, w) = scale\_w × scale\_a × Σc,r,s (QW(k, c, r, s) - zp\_w) × (QA(n, c, h + r, w + s) - zp\_a) **(1)** There are two ways where we can go from here. First, break it down into simpler operators and write schedules for each one of them. Second, keep this form as it is and optimize the schedule in one go. # Option 1 - Simplify the compute into 4 operations Let’s look at the terms that are inside the Σc,r,s in equation (1). Let’s also call this operator `q_conv2d`, which takes quantized matrices and their quantization parameters. (We would also need output quantization parameters here, but skipping that detail to stay focussed) > Σc,r,s (QW(k, c, r, s) - zp\_w) × (QA(n, c, h + r, w + s) - zp\_a) > Σc,r,s QW(k, c, r, s) × QA(n, c, h + r, w + s) // Term 1   - Σc,r,s zp\_a × QW(k, c, r, s)                     // Term 2   - Σc,r,s zp\_w × QA(n, c, h + r, w + s)        // Term 3   + Σc,r,s zp\_a × zp\_w                                // Term 4 Now, the whole computation is broken down into 4 terms - Term 1 - Normal conv2d between the two quantized (u)int8 matrices. This is awesome because it allows to reuse the conv2d schedules. For ARM CPU, we can use VMLAL instruction (LLVM already does that). For Intel, we can use VNNI that performs compute on int8 which internally upcasts to int16/int32 whenever required. - Term 2 - Needs new operator. Constant folding gets rid of this - Term 3 - Needs new operator. Sum reduction on input quantized matrix. This might bring some overhead. But the operations are much lower than Term1. - Term 4 - Constant folding. Just few multiplications of scalar values. The way I envision this is that framework parsers will map the qunatized\_conv2d of frameworks to relay q\_conv2d (with quantize/dequantize wherever necessary). Then, a simplify\_inference type pass can break the q\_conv2d into simpler operators. The unknown here is how bad is the 3rd term. # Option 2 - Keep the compute intact In this case, the equation (1) directly goes into the TVM schedule. @FrozenGene mentions that QNNPACK takes this path, which gives this option some force. In addition, @FrozenGene has internally tested this, potentially giving higher performance than TFLite itself. [See this](https://discuss.tvm.ai/t/tf-lite-quantized-conv2d-operator-conversion/2651/6?u=janimesh) for little more details. Other good thing is that we can keep the `q_conv2d` API close to QNNPack, so that we can fallback to QNNPack codegen if necesasry. > Σc,r,s (QW(k, c, r, s) - zp\_w) × (QA(n, c, h + r, w + s) - zp\_a) The problem that I see with Option 2 though is two-fold - It has to upcast the quantized tensor elements from int8 to int32 to perform the subtraction from zero\_point first. Once it has been upcasted to int32, we can’t use Intel VNNI or ARM VMLAL. It might be possible to somehow undo this in schedule and make schedule look like Option 1, but that seems hacky. - TF Lite uses GemmLowP for implementing the conv. I have not delved into many details, but it performs im2col transformation, followed by matrix multiplication. The im2col transformation, being memory bandwidth bound, has been shown to be not the best option for many cases. So, my hunch is that, for end-to-end, we might be able to perform better than GemmLowP with our layout transformation passes. # Option 3 - Best of both worlds We can keep Option 1 as a Relay optimization pass, that a user can control by Relay BuildConfig. In this manner, we can keep both options. * * * _Note - I have deliberately skipped scale\_w, scale\_a and output quantization parameter handling. It will make discussion difficult. Will tackle them in a separate thread if necessary._ ---
**Author:** ![FrozenGene](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/frozengene/32/1114_2.png) [@FrozenGene](https://discuss.tvm.apache.org/u/FrozenGene)\ **Post date:** [May 21, 2019, 5:48pm UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/9 "2019-05-21T17:48:56Z")
I haven’t seen detail. But I think @ajtulloch should have interest. ---
**Author:** ![FrozenGene](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/frozengene/32/1114_2.png) [@FrozenGene](https://discuss.tvm.apache.org/u/FrozenGene)\ **Post date:** [May 22, 2019, 2:13am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/10 "2019-05-22T02:13:39Z")
Also @merrymercy. I will read this post today. ---
**Author:** ![FrozenGene](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/frozengene/32/1114_2.png) [@FrozenGene](https://discuss.tvm.apache.org/u/FrozenGene)\ **Post date:** [May 22, 2019, 9:49am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/11 "2019-05-22T09:49:46Z")
In fact, on ARM CPU, if want to produce VMLAL, normal way can not complete. i.e. Option 1 / 2 both can not do. Because ARM CPU only support int16-\>int32 VMLAL. So we have to tensorize. i.e. LOAD 8 elements U8 and convert to INT16, SUB zero\_point, then compute, finally we can leverage VMLAL. Option 1 maybe suit for VNNI instruction on Intel CPU. Maybe I prefer Option 2. i.e. we provide standard / normal spatial\_pack schedule on ARM CPU / Intel CPU, and . on generic target (i.e. nn.py), we provide `q_conv2d`'s compute in naive way. Spatial pack schedule performs well in fact according to our test and implement it very easily. But if you want to improve, you could write tensorize version for your target. For example, to produce ARM CPU’s VMLAL / Intel’s VNNI. ---
**Author:** ![janimesh](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/j/b5e925/32.png) [@janimesh](https://discuss.tvm.apache.org/u/janimesh)\ **Post date:** [May 22, 2019, 3:51pm UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/12 "2019-05-22T15:51:10Z")
> **[TF Lite quantized conv2d operator conversion](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/11)** > > In fact, on ARM CPU, if want to produce VMLAL, normal way can not complete. i.e. Option 1 / 2 both can not do. Because ARM CPU only support int16-\>int32 VMLAL. So we have to tensorize. i.e. LOAD 8 elements U8 and convert to INT16, SUB... How about we prototype then? I remember seeing VMLAL instruction long time back just by using LLVM on int8 instructions (no tensorize call). But I might be wrong. For Intel, we do have to use tensorize as LLVM is not strong enough there. I can write Term 1 and Term 3 python in Option 1 and get some performance numbers. Though these will be unoptimized, hopefully we can get some more clarity. What do you say? ---
**Author:** ![jnorwood](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/jnorwood/32/655_2.png) [@jnorwood](https://discuss.tvm.apache.org/u/jnorwood)\ **Post date:** [May 22, 2019, 6:51pm UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/13 "2019-05-22T18:51:13Z")
consider handling symmetric int8 kernel (fixed 0 offset) and uint8 with 0 offset as specializations. terms disappear with these. maybe your constant folding handles that already… also, I believe tflite implementations perform quantized downscale to uint8 feature data between layers. In tflite these are implemented by integer mpy and downshift operations. Some target devices don’t have fp, and especially not dp, so maybe consider supporting the integer implementation as an option. ---
**Author:** ![janimesh](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/j/b5e925/32.png) [@janimesh](https://discuss.tvm.apache.org/u/janimesh)\ **Post date:** [May 22, 2019, 7:52pm UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/14 "2019-05-22T19:52:46Z")
@jnorwoodBoth very good points. Let me answer them one by one If we look at the following equation. > Σc,r,s QW(k, c, r, s) × QA(n, c, h + r, w + s) // Term 1   - Σc,r,s zp\_a × QW(k, c, r, s)                     // Term 2   - Σc,r,s zp\_w × QA(n, c, h + r, w + s)        // Term 3   + Σc,r,s zp\_a × zp\_w                                // Term 4 With both offsets set to 0, we are only left with Term 1. This is normal conv2d of the original quantized matrices. So, I believe that if we design the q\_conv2d API correctly, then setting the zero\_point from outside to zero should satisfy. I am sure that even Option 2 can easily account for this in the schedule and not perform redundant SUB. But, good to keep that in mind. Secondly, yes I agree we need to take care of what is supported in HW. We currently tackle each target separately, so we should be able to keep that differentiation. I agree that rounding and shifts require much more involvement as far as the assembly instructions go. There are lots of tricks there and we can mess up easily. ---
**Author:** ![FrozenGene](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/frozengene/32/1114_2.png) [@FrozenGene](https://discuss.tvm.apache.org/u/FrozenGene)\ **Post date:** [May 23, 2019, 3:05am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/15 "2019-05-23T03:05:48Z")
I think you maybe remember @ziheng’s implementation. His original implementation also make data / weight’s data type be int16, not int8. also confirmed with @merrymercy before. For the option 1, I think we have one thing we must consider. i.e. padding. In TFLite’s quantization model, the pad value should be input\_zero\_point, not 0. However, current normal conv2d doesn’t have one parameter to accept input\_zero\_point. @jackwish should also have interest. ---
**Author:** ![lixiaoquan](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/lixiaoquan/32/245_2.png) [@lixiaoquan](https://discuss.tvm.apache.org/u/lixiaoquan)\ **Post date:** [May 23, 2019, 3:35am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/16 "2019-05-23T03:35:16Z")
There are lots of quantized operations in TFLite, do you plan to implement each one needed as a new op in TVM , like q\_conv2d? I think it is not easy to break those tflite op into smaller tvm ops. ---
**Author:** ![FrozenGene](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/frozengene/32/1114_2.png) [@FrozenGene](https://discuss.tvm.apache.org/u/FrozenGene)\ **Post date:** [May 23, 2019, 4:35am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/17 "2019-05-23T04:35:09Z")
Right. Because quantization computation is not the same as normal computation. Such as q\_add / q\_pool and so on. ---
**Author:** ![janimesh](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/j/b5e925/32.png) [@janimesh](https://discuss.tvm.apache.org/u/janimesh)\ **Post date:** [May 23, 2019, 5:49am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/18 "2019-05-23T05:49:43Z")
So, currently for FP32, we spend lot of time writing schedules for important operators like conv2d, fully connected. The point is that if an operator takes lot of execution time, it is worth it to focus efforts on that. I am using the same principle here. So, we will have to send time on q\_conv2d. As far as writing schedules go, both options might need tensorize work. I agree that for operators like Relu and Pool, we might want to use the Option 1 as default, and they might give good enough performance. ---
**Author:** ![janimesh](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/j/b5e925/32.png) [@janimesh](https://discuss.tvm.apache.org/u/janimesh)\ **Post date:** [May 23, 2019, 5:56am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/19 "2019-05-23T05:56:44Z")
@FrozenGene I don’t understand the int16 and int8 comment. I was looking at this [https://github.com/dmlc/tvm/blob/master/tests/python/unittest/test\_codegen\_arm.py#L45](https://github.com/dmlc/tvm/blob/master/tests/python/unittest/test_codegen_arm.py#L45) This one the matrices are int8. But it does have VMLAL instruction without any tensorization. Is it possible to provide more context here? Padding zero point instead of 0 is good point to remember. Can we do explicit padding before? Maybe explicit pad operator before conv is the right way to go? If I understand, this has to be handled in both options, and does not cause any differentiation point in choosing one option. ---
**Author:** ![jackwish](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/jackwish/32/658_2.png) [@jackwish](https://discuss.tvm.apache.org/u/jackwish)\ **Post date:** [May 23, 2019, 6:06am UTC](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651/20 "2019-05-23T06:06:35Z")
Thanks @FrozenGene for ping me. It’s very glad to see some many inspiring discussions on quantization. I have co-worked with @FrozenGene in the past several months on quantization regarding TVM, and especially on precision/accuracy maintenance of quantizing FP32/INT8 and performance design in INT8 quantization computing, I’d like to share some of my opinions. Before going further, I’d like to say that the design decision depends on **What you want most from quantization** : 1. **Performance** : to exhaust out every drop of the device capability. 2. **Development efficiency** : to reuse/share the computing/scheduling of code/operator that TVM already have. 3. **Quantizing technology** : try out some techniques which can convert/compute INT8 without significant accuracy impact. ## Performance As we all know that quantization brings two advantages: less disk/memory usage, faster inference. As we are talking about INT8 here, the first advantage is not a problem. We internally want performance improvement most which is our value. Initially, we observed 1.5x - 2x performance gap between our convolution schedule and [QNNPACK](https://code.fb.com/ml-applications/qnnpack/) which shows the most powerful performance (btw, QNNPACK shares author with [NNPACK](https://github.com/Maratyszcza/NNPACK)) AFAIK. But, where the INT8 performance comes from? The memory bandwidth or the computing effort? Our experience shows that the memory bandwidth really matters (as we are not going to reduce the multiplication in quantization context like [Winograd](https://en.wikipedia.org/wiki/Coppersmith%E2%80%93Winograd_algorithm) or [Strassen](https://en.wikipedia.org/wiki/Strassen_algorithm) algorithm). Eventually, we choose to follow the QNNPACK approach, of which the computing steps include: 1. Accumulated multiplication with zero point and bias. 2. Requantization the accumulated result (INT32) to (INT8). QNNPACK fuse them as one, which reads INT8 and writes INT8. If we break these two steps into two operators, the first reads INT8 writes INT32 into memory and the second reads INT32 and writes INT8. In our test, this approach showed significant performance drop. (I’d like to make it clear that we have tensorized _step 1_ which may prevent _step 2_ from fusion.) As soon as we merged them into one in tensorize micro kernel, we got basically same performance as QNNPACK. The difference here is if there is INT32 intermedia memory access in the operator, if the computing is merged, the INT32 intermedia result (the accumulated result) can serve in registers. Regarding @janimesh 's proposals, there will be at least two INT32 intermediate result memory access (if we ignore the graph fusion issue) which may bring effort to optimize. So, according to our experience, _option 2_ is more likely to get performance at QNNPACK level. (Note that, tensorize micro kernel looks to be a must for performance in such scenario due to the limitation of TVM codegen.) There could be other design that has more outstanding performance, but we have not gone that far. ## Development efficiency As @FrozenGene has addressed, we proposed new quantization-dedicated operators such as `q_conv2d` (NHWC layout) which has very different schedule when compared to normal conv2d. We eventually chose to add these operators because we failed to get sound performance in _modified_ `conv2d`. As we need brand new schedule for (basically) all new operators, there is much work lies ahead. So, I think one potential advantage of _option 1_ is it can reuse the existing schedule by simply modify `conv2d`. This is more easy to get the work done, though we may not get an _extremely_ good performance - which maybe fine for most people. ## Quantization technology I think many people are interested in trying out quantization technique without the dependency to TensorFlow/TFLite. This is out of this topic, let’s simply ignore it… ## One more thing Please always keep in mind that, the **Requantize** which converts INT32 accumulated result into INT8 is needed, and should be part of quantization version conv2d. So, to summarize, regarding @janimesh 's proposals, I think _option 1_ may get performance similar to TFLite, while _option 2_ is more capable of enabling powerful tensorize design. Thanks. [Next page](https://discuss.tvm.apache.org/t/tf-lite-quantized-conv2d-operator-conversion/2651.md?page=2)