[BUG] Performance drop with batch and opt_level=3

Troubleshooting
#1

When autotuning with batch_size more than 1 and opt_level=3 with mxnet model multiple outputs, I encountered a performance drop in some of the output.
opt_level 0, 1 and 2 seems to be fine

The performance of tvm model batch_size=2, opt_level=3

Similarity Score for output 1 : 0.88 
Similarity Score for output 2 : 0.55 
Similarity Score for output 3 : 1.00 
Similarity Score for output 4 : 0.86 
Similarity Score for output 5 : 1.00
2 Likes
#2

I’m facing the same issue and I can reproduce it with a retinaface model (this model).

When batch size is 1, outputs are correct.

When batch size is 2 or bigger and opt_level is 1 or 2, outputs are correct.

When batch size is 2 or bigger and opt_level is 3, some outputs are wrong/different when compared to the original model.

Is this a bug? Any idea why this happens? @tqchen

#3

hmm maybe a pass or compute/schedule that kicks in only when opt_level = 3 is assuming that a batch size is one. If you have a repro, please open an issue and I can take a look.

To be clear, is this an accuracy or performance problem? Maybe both?

#4

It’s an accuracy issue for sure. I haven’t tested performance yet, but I will.

I’ll open an issue on github then, thanks!

#5

ah “performance” @guyzsarun mentioned seems to mean “accuracy”. Ok.

#6

Thanks @gasgallo , can you try to optionally disable passes that in tvm/build_module.cc at main · apache/tvm · GitHub and see what passes causes the accuracy drop?

One possible pass could be FoldScaleAxis

1 Like
#7

Please refer to Accuracy drop when use batch and opt_level=3 · Issue #7563 · apache/tvm (github.com)

1 Like