Accuracy loss after Auto-tuning

Questions
1

Hello,

I have optimised inceptionv3 model using auto-tuner according to below guide,
https://docs.tvm.ai/tutorials/autotvm/tune_nnvm_mobile_gpu.html#sphx-glr-tutorials-autotvm-tune-nnvm-mobile-gpu-py

auto-tuner parameters I used,
tuner : xgb
n_trial : 50
early_stopping : 450
timeout : 100

After auto-tuning, inference performance shown about 34% faster than before(used pre-trained model). However, accuracy was lower like below,

Before auto-tuning(input image is elephant),
predicted is 387(Indian elephant)

After auto-tuning(input image is same)
predicted is 623(lawn mower)

The accuracy should be no change after auto-tuning? Or the accuracy can be lower?
If the accuracy cannot be lower then is there something I’m missing?

Thanks,
Inki Dae

2

Hello,

Seems the tutorial code uses not trained model data. So I modified the tutorial code so that pre-trainded net and params are delivered to extract_from_graph function. As a result, accuracy is same but it shown inference performance slower than before.

Below is the result.

Before auto-tuning,

ellapsed time is 1s + 714 ms
ellapsed time is 1s + 710 ms
ellapsed time is 1s + 724 ms
ellapsed time is 1s + 729 ms
predicted is 387

After auto-tuning,

ellapsed time is 1s + 748 ms
ellapsed time is 1s + 784 ms
ellapsed time is 1s + 784 ms
ellapsed time is 1s + 775 ms
predicted is 387

And tuning logs,

Tuning…
[Task 1/44] Current/Best: 0.26/ 0.68 GFLOPS | Progress: (50/50) | 53.61 s Done.
[Task 2/44] Current/Best: 0.26/ 3.96 GFLOPS | Progress: (50/50) | 380.44 s Done.
[Task 3/44] Current/Best: 0.19/ 5.56 GFLOPS | Progress: (50/50) | 680.64 s Done.
[Task 4/44] Current/Best: 0.44/ 5.36 GFLOPS | Progress: (50/50) | 645.94 s Done.
[Task 5/44] Current/Best: 0.30/ 5.61 GFLOPS | Progress: (50/50) | 242.78 s Done.
[Task 6/44] Current/Best: 0.86/ 5.09 GFLOPS | Progress: (50/50) | 148.33 s Done.
[Task 7/44] Current/Best: 0.14/ 3.41 GFLOPS | Progress: (50/50) | 721.74 s Done.
[Task 8/44] Current/Best: 0.20/ 7.40 GFLOPS | Progress: (50/50) | 530.31 s Done.
[Task 9/44] Current/Best: 0.15/ 7.12 GFLOPS | Progress: (50/50) | 276.77 s Done.
[Task 10/44] Current/Best: 0.09/ 5.04 GFLOPS | Progress: (50/50) | 315.45 s Done.
[Task 11/44] Current/Best: 2.78/ 5.57 GFLOPS | Progress: (50/50) | 84.55 s Done.
[Task 12/44] Current/Best: 0.29/ 5.36 GFLOPS | Progress: (50/50) | 120.34 s Done.
[Task 13/44] Current/Best: 1.05/ 3.10 GFLOPS | Progress: (50/50) | 157.78 s Done.
[Task 14/44] Current/Best: 2.43/ 2.68 GFLOPS | Progress: (50/50) | 116.74 s Done.
[Task 15/44] Current/Best: 1.06/ 5.93 GFLOPS | Progress: (50/50) | 210.44 s Done.
[Task 16/44] Current/Best: 0.50/ 5.71 GFLOPS | Progress: (50/50) | 128.61 s Done.
[Task 17/44] Current/Best: 1.69/ 5.80 GFLOPS | Progress: (50/50) | 142.58 s Done.
[Task 18/44] Current/Best: 4.75/ 5.20 GFLOPS | Progress: (50/50) | 156.85 s Done.
[Task 19/44] Current/Best: 4.86/ 6.06 GFLOPS | Progress: (50/50) | 90.19 s Done.
[Task 20/44] Current/Best: 3.27/ 5.58 GFLOPS | Progress: (50/50) | 110.48 s Done.
[Task 21/44] Current/Best: 2.09/ 5.40 GFLOPS | Progress: (50/50) | 76.31 s Done.
[Task 22/44] Current/Best: 5.29/ 5.78 GFLOPS | Progress: (50/50) | 89.44 s Done.
[Task 23/44] Current/Best: 5.56/ 5.64 GFLOPS | Progress: (50/50) | 77.05 s Done.
[Task 24/44] Current/Best: 0.23/ 6.04 GFLOPS | Progress: (50/50) | 112.55 s Done.
[Task 25/44] Current/Best: 6.09/ 6.11 GFLOPS | Progress: (50/50) | 74.41 s Done.
[Task 26/44] Current/Best: 4.83/ 5.72 GFLOPS | Progress: (50/50) | 118.37 s Done.
[Task 27/44] Current/Best: 6.03/ 6.18 GFLOPS | Progress: (50/50) | 73.19 s Done.
[Task 28/44] Current/Best: 4.21/ 5.28 GFLOPS | Progress: (50/50) | 72.11 s Done.
[Task 29/44] Current/Best: 2.78/ 3.08 GFLOPS | Progress: (50/50) | 473.09 s Done.
[Task 30/44] Current/Best: 2.72/ 2.74 GFLOPS | Progress: (50/50) | 36.34 s Done.
[Task 31/44] Current/Best: 0.00/ 5.25 GFLOPS | Progress: (50/50) | 154.21 s Done.
[Task 32/44] Current/Best: 4.63/ 4.68 GFLOPS | Progress: (50/50) | 66.85 s Done.
[Task 33/44] Current/Best: 4.87/ 5.32 GFLOPS | Progress: (50/50) | 65.54 s Done.
[Task 34/44] Current/Best: 3.15/ 3.82 GFLOPS | Progress: (50/50) | 60.38 s Done.
[Task 35/44] Current/Best: 5.58/ 5.58 GFLOPS | Progress: (50/50) | 107.23 s Done.
[Task 36/44] Current/Best: 5.40/ 5.70 GFLOPS | Progress: (50/50) | 96.80 s Done.
[Task 37/44] Current/Best: 5.92/ 6.97 GFLOPS | Progress: (50/50) | 172.39 s Done.
[Task 38/44] Current/Best: 3.53/ 5.04 GFLOPS | Progress: (50/50) | 66.70 s Done.
[Task 39/44] Current/Best: 0.55/ 5.47 GFLOPS | Progress: (50/50) | 72.93 s Done.
[Task 40/44] Current/Best: 0.00/ 0.00 GFLOPS | Progress: (50/50) | 4.17 s Done.
[Task 41/44] Current/Best: 2.75/ 3.98 GFLOPS | Progress: (50/50) | 88.97 s Done.
[Task 42/44] Current/Best: 2.68/ 4.80 GFLOPS | Progress: (50/50) | 431.11 s Done.
[Task 43/44] Current/Best: 3.90/ 4.53 GFLOPS | Progress: (50/50) | 243.39 s Done.
[Task 44/44] Current/Best: 1.16/ 3.69 GFLOPS | Progress: (50/50) | 97.21 s Done.

According to above logs, I think it should shown better performance than before.

Thanks,
Inki Dae

3

50 trials is usually not enough to achieve reasonable performance when autotuning. In fact, if you did not change the xgboost tuner settings from their defaults, 50 trials will do nothing more than random search for each workload. Try using a larger number of trials (e.g., 1000).

1 Like
4

Thanks for reply.

Thanks,
Inki Dae