VTA Setup Issue - program_fpga() API fails

Hi, I’m trying to get through this tvm tutorial, except on a ZC706 instead of PYNQ board. I understand that the ZC706 isn’t technically supported by the VTA project, but given that they are both built around Zynq-7000 family FPGAs, I figured I might be able to get it running. I successfully compiled a PYNQ image (v2.5.1) for the ZC706 and was able to build the VTA runtime and RPC server on it, as instructed in that tutorial. However, when I try to run the test_program_rpc.py script, it fails with the below error that I can’t quite make sense of, and that I can’t seem to find anywhere on the forums. What might be going wrong?

On the host side:

$ python3 vta/tests/python/pynq/test_program_rpc.py 
Traceback (most recent call last):

  File "vta/tests/python/pynq/test_program_rpc.py", line 44, in <module>
    program_rpc_bitstream()

  File "vta/tests/python/pynq/test_program_rpc.py", line 35, in program_rpc_bitstream
    program_fpga(remote, path)

  File "/home/adam/tvm/tvm/vta/python/vta/rpc_client.py", line 59, in program_fpga
    fprogram(os.path.basename(bitstream))

  File "/home/adam/tvm/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 216, in __call__
    raise get_last_ffi_error()

tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (8) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCServerLoop(int)+0x6b) [0xb59cd620]
  [bt] (7) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::ServerLoop()+0xa9) [0xb59c6e3e]
  [bt] (6) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::HandleUntilReturnEvent(tvm::runtime::TVMRetValue*, bool, tvm::runtime::PackedFunc const*)+0xfb) [0xb59c6c48]
  [bt] (5) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandleNextEvent(tvm::runtime::TVMRetValue*, bool, tvm::runtime::PackedFunc const*)+0x175) [0xb59caa4a]
  [bt] (4) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandleRecvPackedSeqArg()+0x23f) [0xb59ca5e8]
  [bt] (3) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::SwitchToState(tvm::runtime::RPCSession::EventHandler::State)+0x1eb) [0xb59c9960]
  [bt] (2) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::EventHandler::HandlePackedCall()+0x417) [0xb59c5fd4]
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::PackedFunc::CallPacked(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const+0x33) [0xb59c7e30]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x231cc) [0xb59741cc]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 78, in program_fpga
    program_bitstream.bitstream_program(env.TARGET, path)
  File "/home/xilinx/tvm/vta/python/vta/program_bitstream.py", line 59, in bitstream_program
    pynq_bitstream_program(bitstream)
  File "/home/xilinx/tvm/vta/python/vta/program_bitstream.py", line 49, in pynq_bitstream_program
    bitstream.download()
  File "/usr/local/lib/python3.6/dist-packages/pynq/bitstream.py", line 154, in download
    self.device.download(self, parser)
  File "/usr/local/lib/python3.6/dist-packages/pynq/pl_server/device.py", line 627, in download
    fd.write(bitstream.binfile_name)
TVMError: Except caught from RPC call: TimeoutError: [Errno 110] Connection timed out
During handling of the above exception, another exception occurred:

TimeoutError: [Errno 110] Connection timed out

On the ZC706 (two attempts):

 $ sudo ./apps/vta_rpc/start_rpc_server.sh 
[sudo] password for xilinx: 
INFO:RPCServer:bind to 0.0.0.0:9091
INFO:RPCServer:connection from ('192.168.1.123', 51114)
INFO:root:Rebuild runtime:
 output=/home/xilinx/tvm/vta/python/vta/../../../build/libvta.so,
 cflags=-O2
	-std=c++14
	-I/home/xilinx/tvm/include
	-I/home/xilinx/tvm/3rdparty/vta-hw/include
	-I/home/xilinx/tvm/3rdparty/dlpack/include
	-I/home/xilinx/tvm/3rdparty/dmlc-core/include
	-DVTA_TARGET=pynq
	-DVTA_HW_VER=0.0.1
	-DVTA_LOG_INP_WIDTH=3
	-DVTA_LOG_WGT_WIDTH=3
	-DVTA_LOG_ACC_WIDTH=5
	-DVTA_LOG_BATCH=0
	-DVTA_LOG_BLOCK=4
	-DVTA_LOG_UOP_BUFF_SIZE=15
	-DVTA_LOG_INP_BUFF_SIZE=15
	-DVTA_LOG_WGT_BUFF_SIZE=18
	-DVTA_LOG_ACC_BUFF_SIZE=17
	-DVTA_LOG_BLOCK_IN=4
	-DVTA_LOG_BLOCK_OUT=4
	-DVTA_LOG_OUT_WIDTH=3
	-DVTA_LOG_OUT_BUFF_SIZE=15
	-DVTA_LOG_BUS_WIDTH=6
	-DVTA_IP_REG_MAP_RANGE=0x1000
	-DVTA_FETCH_ADDR=0x43C00000
	-DVTA_LOAD_ADDR=0x43C01000
	-DVTA_COMPUTE_ADDR=0x43C02000
	-DVTA_STORE_ADDR=0x43C03000
	-DVTA_FETCH_INSN_COUNT_OFFSET=16
	-DVTA_FETCH_INSN_ADDR_OFFSET=24
	-DVTA_LOAD_INP_ADDR_OFFSET=16
	-DVTA_LOAD_WGT_ADDR_OFFSET=24
	-DVTA_COMPUTE_DONE_WR_OFFSET=16
	-DVTA_COMPUTE_DONE_RD_OFFSET=24
	-DVTA_COMPUTE_UOP_ADDR_OFFSET=32
	-DVTA_COMPUTE_BIAS_ADDR_OFFSET=40
	-DVTA_STORE_OUT_ADDR_OFFSET=16
	-DVTA_COHERENT_ACCESSES=true,
 source=/home/xilinx/tvm/3rdparty/vta-hw/src/pynq/pynq_driver.cc,
 ldflags=-L/usr/lib
	-l:libcma.so
INFO:root:Generating grammar tables from /usr/lib/python3.6/lib2to3/Grammar.txt
INFO:root:Generating grammar tables from /usr/lib/python3.6/lib2to3/PatternGrammar.txt
INFO:RPCServer:Finish serving ('192.168.1.123', 51114)
INFO:RPCServer:connection from ('192.168.1.123', 51260)
INFO:root:Generating grammar tables from /usr/lib/python3.6/lib2to3/Grammar.txt
INFO:root:Generating grammar tables from /usr/lib/python3.6/lib2to3/PatternGrammar.txt
INFO:RPCServer:Finish serving ('192.168.1.123', 51260)

I am having a very similar error. In my case I have build PYNQ v2.5.1 in a Zybo board. In the host side I get:

~/tvm$ python3 vta/tests/python/pynq/test_program_rpc.py
Traceback (most recent call last):
  File "vta/tests/python/pynq/test_program_rpc.py", line 46, in <module>
    program_rpc_bitstream()
  File "vta/tests/python/pynq/test_program_rpc.py", line 36, in program_rpc_bitstream
    program_fpga(remote, path)
  File "/home/david/tvm/vta/python/vta/rpc_client.py", line 60, in program_fpga
    fprogram(os.path.basename(bitstream))
  File "/home/david/.local/lib/python3.8/site-packages/tvm-0.8.dev206+ga7b22ab9b-py3.8-linux-x86_64.egg/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
    raise get_last_ffi_error()
tvm.error.RPCError: Traceback (most recent call last):
  [bt] (8) /home/david/.local/lib/python3.8/site-packages/tvm-0.8.dev206+ga7b22ab9b-py3.8-linux-x86_64.egg/tvm/libtvm.so(TVMFuncCall+0x63) [0x7fd2b76f1033]
  [bt] (7) /home/david/.local/lib/python3.8/site-packages/tvm-0.8.dev206+ga7b22ab9b-py3.8-linux-x86_64.egg/tvm/libtvm.so(std::_Function_handler<void (tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*), tvm::runtime::RPCModuleNode::WrapRemoteFunc(void*)::{lambda(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*)#1}>::_M_invoke(std::_Any_data const&, tvm::runtime::TVMArgs&&, tvm::runtime::TVMRetValue*&&)+0x37) [0x7fd2b775fce7]
  [bt] (6) /home/david/.local/lib/python3.8/site-packages/tvm-0.8.dev206+ga7b22ab9b-py3.8-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::RPCWrappedFunc::operator()(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const+0x4ba) [0x7fd2b775f72a]
  [bt] (5) /home/david/.local/lib/python3.8/site-packages/tvm-0.8.dev206+ga7b22ab9b-py3.8-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::RPCClientSession::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)> const&)+0x57) [0x7fd2b77548b7]
  [bt] (4) /home/david/.local/lib/python3.8/site-packages/tvm-0.8.dev206+ga7b22ab9b-py3.8-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::RPCEndpoint::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)>)+0x381) [0x7fd2b7749391]
  [bt] (3) /home/david/.local/lib/python3.8/site-packages/tvm-0.8.dev206+ga7b22ab9b-py3.8-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::RPCEndpoint::HandleUntilReturnEvent(bool, std::function<void (tvm::runtime::TVMArgs)>)+0x2d5) [0x7fd2b7748065]
  [bt] (2) /home/david/.local/lib/python3.8/site-packages/tvm-0.8.dev206+ga7b22ab9b-py3.8-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::RPCEndpoint::EventHandler::HandleNextEvent(bool, bool, std::function<void (tvm::runtime::TVMArgs)>)+0x459) [0x7fd2b7754689]
  [bt] (1) /home/david/.local/lib/python3.8/site-packages/tvm-0.8.dev206+ga7b22ab9b-py3.8-linux-x86_64.egg/tvm/libtvm.so(tvm::runtime::RPCEndpoint::EventHandler::HandleReturn(tvm::runtime::RPCCode, std::function<void (tvm::runtime::TVMArgs)>)+0x13c) [0x7fd2b774f47c]
  [bt] (0) /home/david/.local/lib/python3.8/site-packages/tvm-0.8.dev206+ga7b22ab9b-py3.8-linux-x86_64.egg/tvm/libtvm.so(+0x13c7f28) [0x7fd2b7745f28]
  [bt] (8) /home/xilinx/tvm/build/libtvm_runtime.so(+0x8cb8e) [0xb5922b8e]
  [bt] (7) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCServerLoop(int)+0x6b) [0xb5922424]
  [bt] (6) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::ServerLoop()+0x91) [0xb590f2b6]
  [bt] (5) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::HandleUntilReturnEvent(bool, std::function<void (tvm::runtime::TVMArgs)>)+0x1ed) [0xb590eeea]
  [bt] (4) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::HandleNextEvent(bool, bool, std::function<void (tvm::runtime::TVMArgs)>)+0x89) [0xb59151da]
  [bt] (3) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::EventHandler::HandleProcessPacket(std::function<void (tvm::runtime::TVMArgs)>)+0x115) [0xb59150e2]
  [bt] (2) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCSession::AsyncCallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::RPCCode, tvm::runtime::TVMArgs)>)+0x39) [0xb59213b2]
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::LocalSession::CallFunc(void*, TVMValue const*, int const*, int, std::function<void (tvm::runtime::TVMArgs)> const&)+0x39) [0xb591852a]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x2c768) [0xb58c2768]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 81, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 78, in program_fpga
    program_bitstream.bitstream_program(env.TARGET, path)
  File "/home/xilinx/tvm/vta/python/vta/program_bitstream.py", line 62, in bitstream_program
    pynq_bitstream_program(bitstream)
  File "/home/xilinx/tvm/vta/python/vta/program_bitstream.py", line 49, in pynq_bitstream_program
    bitstream.download()
  File "/usr/local/lib/python3.6/dist-packages/pynq/bitstream.py", line 154, in download
    self.device.download(self, parser)
  File "/usr/local/lib/python3.6/dist-packages/pynq/pl_server/device.py", line 627, in download
    fd.write(bitstream.binfile_name)
  File "/home/david/tvm/src/runtime/rpc/rpc_endpoint.cc", line 378
RPCError: Error caught from RPC call:
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

TimeoutError: [Errno 110] Connection timed out

And in the RPC server (on Zybo board):

xilinx@pynq:~/tvm$ sudo ./apps/vta_rpc/start_rpc_server.sh
INFO:RPCServer:bind to 0.0.0.0:9091
INFO:RPCServer:connection from ('192.168.2.1', 40954)
INFO:root:Generating grammar tables from /usr/lib/python3.6/lib2to3/Grammar.txt
INFO:root:Generating grammar tables from /usr/lib/python3.6/lib2to3/PatternGrammar.txt
INFO:RPCServer:Finish serving ('192.168.2.1', 40954)

Any progress, solution or clue about this error? Thanks.