# Expected performance for ToMixedPrecision Float16

**URL:** <https://discuss.tvm.apache.org/t/expected-performance-for-tomixedprecision-float16/14428>\
**Category:** Troubleshooting\
**Created:** [February 24, 2023, 9:24am UTC](https://discuss.tvm.apache.org/t/expected-performance-for-tomixedprecision-float16/14428 "2023-02-24T09:24:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wheest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wheest/32/607_2.png) [@Wheest](https://discuss.tvm.apache.org/u/Wheest)\
**Post date:** [February 24, 2023, 9:24am UTC](https://discuss.tvm.apache.org/t/expected-performance-for-tomixedprecision-float16/14428/1 "2023-02-24T09:24:17Z")

</div>

I have been looking into float16 inference in TVM, converting an existing float32 model within TVM, rather than from an external framework.

@AndrewZhaoLuo has [some nice code](https://github.com/AndrewZhaoLuo/TVM-Sandbox/blob/f1f9f698be2b7a8cc5bcf1167d892cd915eb7ce7/fp16_pass/benchmark_fp16.py#L19) that I’ve been using to test FP16.

However, when running on an NVidia AGX Xavier I find that I’m getting a slowdown on both CPU and GPU. I was also testing on an Intel CPU and saw slowdowns too.

My main project I’m working from is a modified TVM v0.8, but I also tested on an unmodified v0.10 and also saw slowdowns.

I’m just looking at CNN models right now.

The [initial PR](https://github.com/apache/tvm/pull/8069) says that speedups were not initially achieved, but this was later edited to suggest show that some speedups happened.

Are my slowdowns (~1.8x on CPU, 1.05x on GPU) expected right now? I’ve found it with fast-math enabled and disabled, and I’m using all of the opts in the linked sandbox test, including `ToMixedPrecision()(mod)`, plus my regular `-o3` compilation. Is that the right approach?

---

<div class="post-metadata">

**Author:** ![masahi](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/masahi/32/74_2.png) [@masahi](https://discuss.tvm.apache.org/u/masahi)\
**Post date:** [February 24, 2023, 11:21am UTC](https://discuss.tvm.apache.org/t/expected-performance-for-tomixedprecision-float16/14428/2 "2023-02-24T11:21:46Z")

</div>

FP16 on x86 is super slow (it’s software-emulated), and on CUDA it’s no faster than fp32 unless you use tensor core.

---

<div class="post-metadata">

**Author:** ![Wheest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wheest/32/607_2.png) [@Wheest](https://discuss.tvm.apache.org/u/Wheest)\
**Post date:** [February 24, 2023, 3:03pm UTC](https://discuss.tvm.apache.org/t/expected-performance-for-tomixedprecision-float16/14428/3 "2023-02-24T15:03:34Z")

</div>

Good to know, thanks! I just wanted to verify that I wasn’t doing anything wrong on my side!