Reference inference time of Resnet-18 on PYNQ

[aleczhanshi]

Hi there,

I try to get a sense of the inference time for Resnet-18 on PYNQ-Z1. As I run the auto-tuning example on my PYNQ board, the fully-tuned version (grid search all schedules) takes 416 ms for the final inference with a single image, and a barely tuned version (grid search with only 1 trial) gives me just slight worse time of 432 ms. As some references, the sample output from the tutorial shows 621.79 ms, and the published VTA paper (see Fig. 8) can achieve ~250 ms.

If I understand correctly, the TimeEvaluator function only measures the time on the board, so I’m trying to understand the variations behind these runs, and a reasonable number to expect for ResNet-18 on PYNQ-Z1. Below is a table that summarises the numbers above.

    Version                   Time  (std dev)  
    fully tuned (ours).       416.24 ms (5.42 ms)
    barely tuned (ours)       432.34 ms (41.63 ms)
    sample output             621.79 ms (0.14 ms)
    VTA paper                 ~250 ms

Any help is appreciated. Thanks!

Zhan

[bai_62868]

Hi @aleczhanshi , Recently, I also want to run ResNet-18 on Pynq-Z1. I meet some problems when I build the main branch tvm codebase. I can import tvm successfully but I can not import vta successfully. Can you share your step-up information such as the version of llvm or the version of tvm codebase you used. Thanks a lot. This is the problem I met when I built the tvm/vta:

RandomModel in VTA?

Hi, when I was installing TVM/VTA on the Pynq-Z1, I used llvm-4.0 and tvm from main branch. I can import tvm successfully but when I import vta, it said AttributeError: module ‘tvm.auto_scheduler._ffi_api’ has no attribute ‘RandomModel’ What should I do to solve this problem? Thanks a lot! [截屏2021-01-27 下午6.46.55]