[VTA] Workaround for Autotuning with One PYNQ Z1 Board

thkim · November 25, 2020, 7:50pm

@hht @isong Thank you for sharing your solution and issues. I’m also trying to run this “tune_relay_vta.py”. After downloading tvm v0.7 released 2020-10-02, I’ve revised “vta_config.json” (TARGET: “sim” -> “pynq” from /3rdparty/vta-hw/config/vta_config.json) and saved a bitstream file “1x16_i8w8a32_15_15_18_17.bit” in the local directory to avoid the wrong link issue. Then, in the build directory, I’ve designated the llvm route of “config.cmake” for the host PC. Likewise, I’ve revised USE_VTA_FPGA OFF -> ON of “config.cmake” for the pynq board. There seemed to be no problems with the device connection through the tracker and the bitstream file connection.

However, when I run the “tune_relay_vta.py” (I’ve commented out the code @hht explained.), it doesn’t increase GFLOPS (always 0.00/ 0.00 GFLOPS). Could you give me some advice? Do I need to change another code?

isong · November 26, 2020, 1:02am

Hi @thkim

If you update your from the above githhub link and my change on the export_library, then let me check few more.

It is not a connection issue, as you see GFLOPS (always 0.00/ 0.00 GFLOPS) message. Does it every finish? How many iteration you’ve set? Are other testing such as this that are not using autotvm works?

Cheers, ISS

hht · November 26, 2020, 1:36am

Check dmesg from your remote device. Sometimes 0.00 GFLOPS means no bitstream writtten to FPGA.

hht · November 26, 2020, 1:39am

Make sure you start remote device server with root because writing bitstream need root authority.

thkim · November 26, 2020, 7:53am

@isong @hht Thank you so much for your reply. I think I found a problem. When I run the tutorial “Deploy Pretrained Vision Model from MxNet on VTA”, I could write a bitstream file to FPGA, and it worked well. I’ve written vta.program_fpga(remote, bitstream=“bitfiles/v4/1x16_i8w8a32_15_15_18_17.bit”) in the sample code. I’ve checked it through dmesg.

However, when I run the tutorial “Auto-tuning a convolutional network on VTA” (tune_relay_vta_with_one_board.py), I couldn’t write a bitstream file to FPGA. During the tuning, it seems to access the program_fpga function of rpc_client.py per 8 iterations and get the bitstream="/home/taeho/.vta_cache/pynq/0_0_1/1x16_i8w8a32_15_15_18_17.bit". When I checked dmesg from my pynq board, it failed to write a bitstream to FPGA. I couldn’t find the exact reason.

For the tutorial “Deploy Pretrained Vision Model from MxNet on VTA”, I’ve run “sudo apps/vta_rpc/start_rpc_server.sh” on pynq. For tune_relay_vta_with_one_board.py, I’ve run “sudo apps/vta_rpc/start_rpc_server_to_tracker.sh” on pynq. @hht I’m not sure if it is correct to start remote device server with root.

Again, thank you so much for your help!

isong · November 26, 2020, 8:16am

Hi @thkim

Did you check if your bit file is at /home/taeho/.vta_cache/pynq/0_0_1/1x16_i8w8a32_15_15_18_17.bit"?

In my case, I copy the bit file to the above dir.

I also use sudo to run tracker

pushd apps/vta_rpc
sudo -E ./start_rpc_server_to_tracker.sh
popd

hht · November 26, 2020, 8:33am

Hi @thkim

I use su to start the tracker and modify the corresponding environment variables.

TVM_HOME="/home/xilinx/tvm"
VTA_HW_PATH="/home/xilinx/tvm/3rdparty/vta-hw"
PYTHONPATH="/home/xilinx/tvm/python:/home/xilinx/tvm/topi/python:/home/xilinx/tvm/vta/python"

hht · November 26, 2020, 8:35am

root@pynq:/home/xilinx# cat tvm/apps/vta_rpc/start_rpc_server_to_tracker.sh 
#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.
PROJROOT="$( cd "$( dirname '${BASH_SOURCE[0]}' )/../../" && pwd )"

# Derive target specified by vta_config.json
VTA_CONFIG=/home/xilinx/tvm/3rdparty/vta-hw/config/vta_config.py
TARGET=$(python ${VTA_CONFIG} --target)

export PYTHONPATH=${PYTHONPATH}:${PROJROOT}/python:${PROJROOT}/vta/python
export PYTHONPATH=${PYTHONPATH}:/home/xilinx/pynq
python3 -m vta.exec.rpc_server --tracker 192.168.0.114:9190 --key $TARGET

thkim · November 26, 2020, 8:44am

@isong @hht Thank you so much for your response. Although I’ve checked a bit file is in /home/taeho/.vta_cache/pynq/0_0_1/1x16_i8w8a32_15_15_18_17.bit, there was a problem. Instead, in the rpc_client.py, I’ve written bitstream = “dir” which is the bit file directory I’ve saved in the local dir. It works now. Thanks! Yes, I also wrote the environment variables like you!

I’ll work on this continuously, so I hope we can share something in the future. Thank you, again

hht · November 26, 2020, 9:07am

@thkim Glad to hear your good news. I have been working to fix the memory leak caused by rpc exception. VTA use CMA memory and CMA is managed by /dev/xlnk. Directly terminating the process will only release heap and stack memory but still cause a memory leak. I used two ways to make the workaround but can’t find an elegant solution. There are a lot of things to do and to discuss.

isong · November 26, 2020, 9:12am

Hi @hht

Interesting, I am interested in what’s your 2 ways fo workarounds? Would you mind sharing them?

Thank you,

@thkim, Glad to hear that you sorted out things.

Cheers,

hht · November 26, 2020, 9:18am

One way is to create a global class record cma alloc and free. When exception occurs, the python-rpc-server firstly provided the C API to invoke xilinx cma_free() to release the cma memory then terminate the process.

hht · November 26, 2020, 9:24am

Another way is to avoid rpc exception and improve tuning efficiency. I replace batch 300 input tensor with batch 1 input tensor. And use simple python script the generate simpler net. I thought that with GEMM batch 300 conv2d could share nearly the same optimal schedule with batch 1 conv2d.

isong · November 26, 2020, 10:47am

Hi @htt

Thank you for sharing your work. It seems the 2nd option made more sense, though it seems to be good to improve rpc exception handling.

Cheers, ISS

thkim · December 4, 2020, 10:31pm

Hi,

I want to check several things.

Does your tutorial work with ‘xgb’ tuner?
I tried to compare non-tuning with tuning and failed to see the improvement for execution time after tuning. Did you see the improvement after tuning?
@isong I’m using one PYNQ-Z1 board. When I ran the tutorial, the mean inference time was 365 ~ 372 ms. Your example shows 69.65 ms. Could you give me some advice to decrease the mean inference time?

Thanks!

hht · December 6, 2020, 4:19am

@thkim

I am not sure, but with random tunner it works.
Because you use the tophub well-tuned params. If you try with fallback config, it will be great improvement.
I use PYNQ Z1 and ZCU104. I think 365ms is reasonable.

isong · December 7, 2020, 1:18am

Hi @thkim

For 1 and 2, I think @hht has already answered.

For 3, I used ultra96, could be the reason why the difference.

isong · December 13, 2020, 10:19am

@hht

I have create a PR that combines your change and my change that is discussed in the thread. I wanted to add you to the PR, but I cannot seem to find you in @ at github id.

hht · December 13, 2020, 12:33pm

@isong

I see you already @ me. Hope this commit can pass.

youxiudeshouyeren · April 21, 2022, 3:33am

Recently, I encountered this problem again. Using a single pynq Z2 will prompt that the device cannot be obtained from remote. My TVM version is v0 8. Do you have any suggestions? Thank you.