[VTA] Recursion error

Julien · July 10, 2020, 7:49pm

I’m following this guide https://tvm.apache.org/docs/vta/install.html to install VTA, but I get I get a recursion error (RecursionError: maximum recursion depth exceeded in comparison) while executing this script in hardware (not in simulation):

python <tvm root>/vta/tests/python/integration/test_benchmark_topi_conv2d.py

Here is the error in detail (host side):

Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=64, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1)
CPU CONV2D TEST PASSED: Time cost = 0.0158294 sec/op, 14.6064 GOPS
Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=128, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.00839859 sec/op, 13.7649 GOPS
Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=128, hkernel=1, wkernel=1, hpad=0, wpad=0, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.00140844 sec/op, 9.12003 GOPS
Conv2DWorkload(batch=1, height=28, width=28, in_filter=128, out_filter=128, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1)
CPU CONV2D TEST PASSED: Time cost = 0.0166927 sec/op, 13.8511 GOPS
Conv2DWorkload(batch=1, height=28, width=28, in_filter=128, out_filter=256, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.00881398 sec/op, 13.1162 GOPS
Conv2DWorkload(batch=1, height=28, width=28, in_filter=128, out_filter=256, hkernel=1, wkernel=1, hpad=0, wpad=0, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.00126221 sec/op, 10.1766 GOPS
Conv2DWorkload(batch=1, height=14, width=14, in_filter=256, out_filter=256, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1)
CPU CONV2D TEST PASSED: Time cost = 0.0138982 sec/op, 16.6361 GOPS
Conv2DWorkload(batch=1, height=14, width=14, in_filter=256, out_filter=512, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.011544 sec/op, 10.0143 GOPS
Conv2DWorkload(batch=1, height=14, width=14, in_filter=256, out_filter=512, hkernel=1, wkernel=1, hpad=0, wpad=0, hstride=2, wstride=2)
CPU CONV2D TEST PASSED: Time cost = 0.00127813 sec/op, 10.0499 GOPS
Conv2DWorkload(batch=1, height=7, width=7, in_filter=512, out_filter=512, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1)
CPU CONV2D TEST PASSED: Time cost = 0.0168659 sec/op, 13.7088 GOPS
Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=64, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1)
Traceback (most recent call last):
  File "../../outils/tvm/vta/tests/python/integration/test_benchmark_topi_conv2d.py", line 257, in <module>
    test_conv2d(device="vta")
  File "../../outils/tvm/vta/tests/python/integration/test_benchmark_topi_conv2d.py", line 253, in test_conv2d
    vta.testing.run(_run)
  File "/home/julien/outils/tvm/vta/python/vta/testing/util.py", line 75, in run
    run_func(env, remote)
  File "../../outils/tvm/vta/tests/python/integration/test_benchmark_topi_conv2d.py", line 252, in _run
    run_conv2d(env, remote, wl, target)
  File "../../outils/tvm/vta/tests/python/integration/test_benchmark_topi_conv2d.py", line 183, in run_conv2d
    data_arr = tvm.nd.array(data_np, ctx)
  File "/home/julien/outils/tvm/python/tvm/runtime/ndarray.py", line 503, in array
    return empty(arr.shape, arr.dtype, ctx).copyfrom(arr)
  File "/home/julien/outils/tvm/python/tvm/runtime/ndarray.py", line 271, in empty
    check_call(_LIB.TVMArrayAlloc(
  File "/home/julien/outils/tvm/python/tvm/_ffi/base.py", line 334, in check_call
    raise get_last_ffi_error()

[...] (error too long)

  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 64, in ext_dev_callback
    return tvm.get_global_func("device_api.ext_dev")()
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 64, in ext_dev_callback
    return tvm.get_global_func("device_api.ext_dev")()
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 64, in ext_dev_callback
    return tvm.get_global_func("device_api.ext_dev")()
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 63, in ext_dev_callback
    load_vta_dll()
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 53, in load_vta_dll
    logging.info("Loading VTA library: %s", dll_path)
  File "/usr/lib/python3.6/logging/__init__.py", line 1900, in info
    root.info(msg, *args, **kwargs)
  File "/usr/lib/python3.6/logging/__init__.py", line 1306, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.6/logging/__init__.py", line 1441, in _log
    exc_info, func, extra, sinfo)
  File "/usr/lib/python3.6/logging/__init__.py", line 1411, in makeRecord
    sinfo)
  File "/usr/lib/python3.6/logging/__init__.py", line 277, in __init__
    if (args and len(args) == 1 and isinstance(args[0], collections.Mapping)
  File "/home/julien/outils/tvm/src/runtime/rpc/rpc_endpoint.cc", line 370
RPCError: Error caught from RPC call:
RecursionError: maximum recursion depth exceeded in comparison

The detailed error (RPC server side):

INFO:RPCServer:bind to 0.0.0.0:9091
INFO:RPCServer:connection from ('192.168.3.102', 48578)
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmp44hzuwsc/conv2d.o
Process Process-1:1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/xilinx/tvm/python/tvm/rpc/server.py", line 85, in _serve_loop
    _ffi_api.ServerLoop(sockfd)
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
AttributeError: Traceback (most recent call last):
  [bt] (4) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (3) /home/xilinx/tvm/build/libtvm_runtime.so(+0xb4d54) [0x7fade35d54]
  [bt] (2) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCServerLoop(int)+0xac) [0x7fade347b4]
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::ServerLoop()+0x1c8) [0x7fade15bb8]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 84, in server_shutdown
    runtime_dll[0].VTARuntimeShutdown()
  File "/usr/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so: undefined symbol: VTARuntimeShutdown
INFO:RPCServer:connection from ('192.168.3.102', 48584)
INFO:root:Program FPGA with 1x16_i8w8a32_15_15_18_17.bit 
INFO:root:Skip reconfig_runtime due to same config.
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:RPCServer:load_module /tmp/tmpk08kcwj4/conv2d.o
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
INFO:root:Loading VTA library: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so
Process Process-1:2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/home/xilinx/tvm/python/tvm/rpc/server.py", line 85, in _serve_loop
    _ffi_api.ServerLoop(sockfd)
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 225, in __call__
    raise get_last_ffi_error()
AttributeError: Traceback (most recent call last):
  [bt] (4) /home/xilinx/tvm/build/libtvm_runtime.so(TVMFuncCall+0x70) [0x7faddb5b90]
  [bt] (3) /home/xilinx/tvm/build/libtvm_runtime.so(+0xb4d54) [0x7fade35d54]
  [bt] (2) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCServerLoop(int)+0xac) [0x7fade347b4]
  [bt] (1) /home/xilinx/tvm/build/libtvm_runtime.so(tvm::runtime::RPCEndpoint::ServerLoop()+0x1c8) [0x7fade15bb8]
  [bt] (0) /home/xilinx/tvm/build/libtvm_runtime.so(+0x31798) [0x7faddb2798]
  File "/home/xilinx/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 78, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/xilinx/tvm/vta/python/vta/exec/rpc_server.py", line 84, in server_shutdown
    runtime_dll[0].VTARuntimeShutdown()
  File "/usr/lib/python3.6/ctypes/__init__.py", line 361, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib/python3.6/ctypes/__init__.py", line 366, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /home/xilinx/tvm/vta/python/vta/../../../build/libvta.so: undefined symbol: VTARuntimeShutdown

I know that we can solve the problem by adding these lines of code, but here it do not work here. Certainly because this code is not offloaded to FPGA board:

import sys
sys.setrecursionlimit(10**6)

Thanks in advance for your help

wowow11111 · July 14, 2020, 4:02am

Same here… any help?

Julien · July 14, 2020, 6:18pm

I tried to change the recursion limit to 1,000,000 (instead of 10,000) in the following files (both host and target):

$TVM_HOME/python/tvm/relay/__init__.py
$TVM_HOME/python/tvm/relay/_parser.py

But still the same error. Any idea?

acapone13 · July 23, 2020, 3:05pm

Any update on this issue?

wowow11111 · July 28, 2020, 10:46am

somehow, I could run without this error. Then this popped out of nowhere again. In just a few seconds. This is what happened.

prev : I could run tutorials without any problems.

I tried to change some parameters (in vta_config.json file) with the configured bitstream file.
Same issue.
I re-changed to the default configuration, and get the same issue with the ones that I could run successfully.

There must be some kind of bug here.

wowow11111 · July 28, 2020, 11:47am

This way, I could make it run again without problem.

re-install from source
make the config file with ‘pynq config’

still have same runtime shutdown issue

change the target to ‘ultra96’ and re make

BTW I am using the ZCU104 board so my target is ultra96. But as soon as I try another configuration bit file or some other attempts using different configs, I go back to the same issue, and returning the setting to the values that used to work, does not work. Then I have to re-install again in that order above.

Haven’t found the solution yet.

acapone13 · July 30, 2020, 7:55pm

I encountered a similar situation. I had everything running flawlessly in my PYNQ boards but when I tried modifying the parameters from the configuration file the remote execution on the FPGA side started failing. I tried re-installing and using the default configuration but It hasn’t worked since then. Haven’t found any solution to the problem either.

Hagoon · August 3, 2020, 8:52am

I also encountered recurrsion error. In my case I just ran vta tutorial - Get Started with VTA - with changing freq and pipeline setting. Although I rebuild both bitstream file in host and cmake in pynq with changed pkg_config.py setting, I cannot run jupytercode with recurrsion error.

wowow11111 · August 6, 2020, 10:34am

If you build vta on device, you’ll find a “libvta.so” file. But as you change small bit on the host side and run any code, 'VTARuntimeShutdown" disappears. I don’t know how this works, but this is the reason of this failure.

For example, if you change LOG_BATCH to 1 (vta/config/vta_config.json) and run any code on pynq, the .so file changes and occurs error. Anyone have any advice?

Best solution for me rn, just make clean on device side and rebuild runtime, vta but this is too big job to do every time I run different code which has different parameters. (Or copy the first default .so file and paste it every time it’s changed, but didn’t try this.)

acapone13 · August 3, 2020, 12:24pm

In my case I haven’t been able to successfully ran any operation on the devices since I changed the configuration parameters. I tried rebuilding and reinstalling everything but I still meet the recursion error.

wowow11111 · August 10, 2020, 1:56am

Any updates on this issue…?

Julien · August 11, 2020, 2:32pm

Few updates on my side. I tried to reset to several older commits, and there is always an issue with RPC server. But it’s never the same issue. Then yesterday I tried the latest commit, but it’s impossible to compile the model on the host side (segmentation fault during quantization). So I’m just working in simulation for now.

wowow11111 · August 12, 2020, 11:14am

Same… just realized today the RPC server connection does not work. For me, the error message is

No module named ‘typed_ast’ which never used to exist,

so I tried installing it with pip and still have the same error.

Groupsun · August 13, 2020, 9:25am

I came across the same problem too, but I try the issue: https://github.com/apache/incubator-tvm/pull/6012

And it seems to work well now.

wowow11111 · August 14, 2020, 6:58am

Thank you for the info. For me, just cleaning and re-making fixes the problem at least for one configuration. Do you have no error after you change some configurations and run any code on the device like PYNQ? The libvta.so file seems to change on my device.

acapone13 · August 17, 2020, 6:30pm

No good news from my side, I keep on getting the RecursionError when I try to execute models with VTA remotely and now I get the same error as in [VTA Tutorial] deploy_classification.py segmentation fault when trying simulation.

jenst · August 24, 2020, 6:48pm

I’m getting this error as well. Any update on this issue?

Julien · August 24, 2020, 8:15pm

No update on my side.

acapone13 · September 5, 2020, 5:51pm

Just tried the latest commit with new VTA updates, unfortunately I keep on getting the same issue with the max recursion depth problem. I tested both in a Pynq-Z2 and an Ultra-96 board with pynq 2.4 and pynq 2.5. Has anybody been able to solve this issue? @thierry would you mind taking a look at this issue?

thierry · September 5, 2020, 10:41pm

I’ve updated the build instructions. There’s an issue with the way the RPC server is being built, and the infinite recursion issue is due to a linker issue.

A temporary solution is to build the runtime on the Pynq, make clean, and build it again. See the PR that was just merged:

I’ll look into why the linking is failing, but for now this should do it. Let me know if you’ve been able to get around the issue.