Test_benchmark_topi_conv2d.py failure caused by AttributeError: Module has no function 'tvm.contrib.vta.init'

torontotong · May 21, 2021, 5:25pm

Hi Everyone,

I am encountering an issue when I try to get tvm host work with my vta-device - A terasic stratix 10 FPGA board. I had vta build on the device side with “set(USE_VTA_FPGA ON)” option. I got rpc_server up running. like below: *terasic@localhost:/media/workspace/tvm$ apps/vta_rpc/start_rpc_server.sh * INFO:RPCServer:bind to 0.0.0.0:9091

Then I try "python vta/tests/python/integration/test_benchmark_topi_conv2d.py " on the host side, The CPU convd2D portion passed test, however, the program stopped when it try to start vta portiaion. Here is lwhat I got on host side:

Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=64, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1) CPU CONV2D TEST PASSED: Time cost = 0.0165433 sec/op, 13.9761 GOPS Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=128, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=2, wstride=2) CPU CONV2D TEST PASSED: Time cost = 0.0087746 sec/op, 13.175 GOPS Conv2DWorkload(batch=1, height=56, width=56, in_filter=64, out_filter=128, hkernel=1, wkernel=1, hpad=0, wpad=0, hstride=2, wstride=2) CPU CONV2D TEST PASSED: Time cost = 0.0013579 sec/op, 9.45951 GOPS Conv2DWorkload(batch=1, height=28, width=28, in_filter=128, out_filter=128, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1) CPU CONV2D TEST PASSED: Time cost = 0.015791 sec/op, 14.6419 GOPS Conv2DWorkload(batch=1, height=28, width=28, in_filter=128, out_filter=256, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=2, wstride=2) CPU CONV2D TEST PASSED: Time cost = 0.00878055 sec/op, 13.1661 GOPS Conv2DWorkload(batch=1, height=28, width=28, in_filter=128, out_filter=256, hkernel=1, wkernel=1, hpad=0, wpad=0, hstride=2, wstride=2) CPU CONV2D TEST PASSED: Time cost = 0.0012507 sec/op, 10.2703 GOPS Conv2DWorkload(batch=1, height=14, width=14, in_filter=256, out_filter=256, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1) CPU CONV2D TEST PASSED: Time cost = 0.0136664 sec/op, 16.9182 GOPS Conv2DWorkload(batch=1, height=14, width=14, in_filter=256, out_filter=512, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=2, wstride=2) CPU CONV2D TEST PASSED: Time cost = 0.0115497 sec/op, 10.0094 GOPS Conv2DWorkload(batch=1, height=14, width=14, in_filter=256, out_filter=512, hkernel=1, wkernel=1, hpad=0, wpad=0, hstride=2, wstride=2) CPU CONV2D TEST PASSED: Time cost = 0.00133672 sec/op, 9.60939 GOPS Conv2DWorkload(batch=1, height=7, width=7, in_filter=512, out_filter=512, hkernel=3, wkernel=3, hpad=1, wpad=1, hstride=1, wstride=1) CPU CONV2D TEST PASSED: Time cost = 0.0167759 sec/op, 13.7823 GOPS Traceback (most recent call last): File “vta/tests/python/integration/test_benchmark_topi_conv2d.py”, line 311, in test_conv2d(device=“vta”) File “vta/tests/python/integration/test_benchmark_topi_conv2d.py”, line 306, in test_conv2d vta.testing.run(_run) File “/media/tliu/ECE699/tvm-repo/dev-tvm-working/vta/python/vta/testing/utils.py”, line 74, in run run_func(env, remote) File “vta/tests/python/integration/test_benchmark_topi_conv2d.py”, line 297, in _run program_fpga(remote, bitstream=None) File “/media/tliu/ECE699/tvm-repo/dev-tvm-working/vta/python/vta/rpc_client.py”, line 64, in program_fpga fprogram = remote.get_function(“tvm.contrib.vta.init”) File “/media/tliu/ECE699/tvm-repo/dev-tvm-working/python/tvm/rpc/client.py”, line 73, in get_function return self._sess.get_function(name) File “/media/tliu/ECE699/tvm-repo/dev-tvm-working/python/tvm/runtime/module.py”, line 91, in get_function raise AttributeError(“Module has no function ‘%s’” % name) AttributeError: Module has no function ‘tvm.contrib.vta.init’

I had a look at the device side, and I have added coresponding entry for ‘tvm.contrib.vta.init’ for my device in rpc_server.py module under tvm/vta/python/vta/exec/ folder.

Anybody had seen this before? I was stopped here for few days, I would appreciate any suggestion, comment, and helps!

Thanks,

Tong

pr3dat0r2k · May 23, 2021, 1:33pm

same problem here on PYNQ Z2. i’m still trying to find where the problem is

pr3dat0r2k · May 24, 2021, 2:16pm

by adding print(“registering %s”%func_name) in register_func’s register func(line 190) in tvm/python/_ffi/registry.py (on board not host machine) and run start_rpc_server.sh again you can see that the functions in rpc_server.py are not registered including tvm.contrib.vta.* which are the most important. you can force them to be loaded: it seems that the server_start() on top is registered so you may move the rest of the functions to the top (instead of being under the server_start() ). i’ll try this tommorow. this might be a bug. btw sry for my bad english. tell me pls if this work.

pr3dat0r2k · May 25, 2021, 8:02am

i have a really bad solution now just move all of the functions in rpc_server.py to python/tvm/rpc/init.py and solve all of the dependencies manually

berank · May 26, 2021, 6:38am

Any updates? I am having the same error.

pr3dat0r2k · May 26, 2021, 10:14am

here is a quick fix. i dont know if it’s legal to upload the init.py i mentioned earlier(because of the license stuff). this is the rpc_server.py file looks like.rest of the functions in this file are moved to the init.py

replace the python/tvm/rpc/init.py with this https://github.com/predator2k/fix_for_vta tell me pls if this is illegal. i know nothing about license.

torontotong · May 26, 2021, 1:49pm

It looks like those modules failed to be registered might be cuase by the python code format indent error of rpc_server.py. Because if I removed tab or space in front of each @tvm.register_func() and def “function name”(): , except

@tvm.register_func(“tvm.rpc.server.start”, override=True)
def server_start():

They were registered properly. I could confirm all modules are registered, see log below:

terasic@localhost:/media/workspace/tvm$ sudo apps/vta_rpc/start_rpc_server.sh
registering diagnostics.override_renderer
registering tvm.default_trace_action
registering target._load_config_dict
registering tvm_callback_rocm_link
registering tvm_callback_rocm_bitcode_path
registering tvm_callback_libdevice_path
registering tvm_callback_sdaccel_compile
registering rpc.test.addone
registering rpc.test.strcat
registering rpc.test.except
registering rpc.test.runtime_str_concat
registering rpc.test.remote_array_func
registering rpc.test.add_to_lhs
registering rpc.test.remote_return_nd
registering rpc.PopenSession
registering tvm_callback_cuda_compile
registering tvm.info.mem.local.inp_buffer
registering tvm.info.mem.local.wgt_buffer
registering tvm.info.mem.local.acc_buffer
registering tvm.rpc.server.start
registering tvm.rpc.server.load_module
registering device_api.ext_dev
registering tvm.contrib.vta.init
registering tvm.rpc.server.shutdown
registering tvm.contrib.vta.reconfig_runtime
registering diagnostics.override_renderer
registering tvm.default_trace_action
registering target._load_config_dict
registering tvm_callback_rocm_link
registering tvm_callback_rocm_bitcode_path
registering tvm_callback_libdevice_path
registering tvm_callback_sdaccel_compile
registering rpc.test.addone
registering rpc.test.strcat
registering rpc.test.except
registering rpc.test.runtime_str_concat
registering rpc.test.remote_array_func
registering rpc.test.add_to_lhs
registering rpc.test.remote_return_nd
registering rpc.PopenSession
INFO:RPCServer:bind to 0.0.0.0:9091

pr3dat0r2k · May 27, 2021, 12:27am

yes but in my rpc test on host the functions in rpc_server.py still cannot be found correctly. only rpc.test related functions can be found so i moved them all to the rpc module

newyt117 · June 11, 2021, 1:48am

yes, I meet the same problem… rpc_server.py the format indent error is fixed, but “Module has no functon ‘tvm.contrib.vta.init’” error still there

torontotong · June 12, 2021, 9:57pm

After some debugging work, step into the source code, I found a solution for this issue. What happened was on the device side, the vta/python/vta/exec/rpc_server.py module has “tvm.contrib.vta.init” and other function implemented, however rpc_server is actually launched an new thread which execute server function from tvm/python/tvm/rpc/server.py file. The server function never call any function implemented in vta/python/vta/exec/rpc_server.py. That’s why when device rpc server receiving runtime request for example ‘tvm.contrib.vta.init’, rpc server reply ‘tvm.contrib.vta.init’” not fount. My solution is add those functions which have been implemented in rpc_server.py’s server_start function to tvm/python/tvm/rpc/server.py file’s _server_env() function. Then everything start to work. functions I added to tvm/python/tvm/rpc/server.py file’s _server_env() function are:

@tvm.register_func("tvm.contrib.vta.init", override=True)
def program_fpga(file_name):
@tvm.register_func("device_api.ext_dev")
def ext_dev_callback():
@tvm.register_func("tvm.contrib.vta.reconfig_runtime", override=True)
def reconfig_runtime(cfg_json):

and their support functions. Hope these helps!

Lytain · July 2, 2021, 2:45am

Hello, I also encountered the same problem, but there are other errors after modification. Can you share the modified “server.py” or “python/tvm/rpc folder”. Thanks!!!

Lytain · July 2, 2021, 12:35pm

Thank you. It’s done!

tqchen · July 30, 2021, 4:23pm

Please see if [VTA] Recover rpc server support by tqchen · Pull Request #8604 · apache/tvm · GitHub fixes the problem

torontotong · September 8, 2021, 5:50pm

Thanks for letting us know the fixing.