[NO LOSS; CLOSED] Has anyone noticed slight accuracy loss in recent tvm?

Questions
1

I ran a keras resnet50 model and its compiled nnvm model on the same 50 images from the ImageNet dataset, and noticed that the recent tvm has a slightly lower accuracy than the old tvm. Results as follows:

tvm commit e986f87:

nnvm, cuda
top1 accuracy :  0.52
top5 accuracy :  0.82

nnvm, llvm
top1 accuracy :  0.533
top5 accuracy :  0.867


tvm commit 2dea429:
nnvm, cuda
top1 accuracy :  0.58
top5 accuracy :  0.84

nnvm, llvm
top1 accuracy :  0.6
top5 accuracy :  0.9


keras:

cuda
top1 accuracy :  0.58
top5 accuracy :  0.84

cpu
top1 accuracy :  0.58
top5 accuracy :  0.84

Hardware: GTX 1060 6GB, Xeon E5-2620 v3. Input_shape: (1, 3, 224, 224)

2

One way to potentially pin down the source of differences is to compare the outputs at each layer.

However, before pinning it down as an accuracy loss, is it possible to evaluate a slightly larger set of images?

3

You’re right. When the number of images increase to 1000, top-1 and top-5 accuracies are the same for nnvm and keras. Both latest tvm and keras have:

top1 accuracy : 0.683
top5 accuracy : 0.893

Thanks.