[AutoTVM] Cannot get remote devices from tracker

Troubleshooting
4

Hi VTA users, @thierry

I am running in the similar issue of running the example. My setup is as follows and runs fine with other VTA code.

  • ultra96
  • pynq version 2.5
  • tvm 5697440cc178e3d901302bc879c1eb506eff8064
  • vta 87ce9acfae550d1a487746e9d06c2e250076e54c
  • host: ubuntu 18.4 (as a side I also tried with Mac, it failed some other fault even it got to this much, and seemingly related to the OS, so I tried ubuntu which went further, close but no cigar).

When I start the tracker I see the devices shown as free and then taken by the autotvm script then showed error. In other words, initial rpc tracker connection was correct.

Also I did changed the check_remote timeout based on this [link](# [solved][AutoTVM] Cannot get remote devices from the tracker), but it didn’t make a difference.

$ python3 ./tune_relay_vta.py 
Extract tasks...
Extracted 10 conv2d tasks:
(1, 14, 14, 256, 512, 1, 1, 0, 0, 2, 2)
(1, 28, 28, 128, 256, 1, 1, 0, 0, 2, 2)
(1, 56, 56, 64, 128, 1, 1, 0, 0, 2, 2)
(1, 56, 56, 64, 64, 3, 3, 1, 1, 1, 1)
(1, 28, 28, 128, 128, 3, 3, 1, 1, 1, 1)
(1, 56, 56, 64, 128, 3, 3, 1, 1, 2, 2)
(1, 14, 14, 256, 256, 3, 3, 1, 1, 1, 1)
(1, 28, 28, 128, 256, 3, 3, 1, 1, 2, 2)
(1, 7, 7, 512, 512, 3, 3, 1, 1, 1, 1)
(1, 14, 14, 256, 512, 3, 3, 1, 1, 2, 2)
Tuning...
Traceback (most recent call last):
  File "./tune_relay_vta.py", line 476, in <module>
    tune_and_evaluate(tuning_option)
  File "./tune_relay_vta.py", line 432, in tune_and_evaluate
    tune_tasks(tasks, **tuning_opt)
  File "./tune_relay_vta.py", line 304, in tune_tasks
    autotvm.callback.log_to_file(tmp_log_file),
  File "//Github/tvm_linux/python/tvm/autotvm/tuner/tuner.py", line 112, in tune
    measure_batch = create_measure_batch(self.task, measure_option)
  File "/Github/tvm_linux/python/tvm/autotvm/measure/measure.py", line 257, in create_measure_batch
    attach_objects = runner.set_task(task)
  File "/Github/tvm_linux/python/tvm/autotvm/measure/measure_methods.py", line 252, in set_task
    "Cannot get remote devices from the tracker. "
RuntimeError: Cannot get remote devices from the tracker. Please check the status of tracker by 'python -m tvm.exec.query_rpc_tracker --port [THE PORT YOU USE]' and make sure you have free devices on the queue status.


^CException ignored in: <module 'threading' from '/usr/lib/python3.6/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 1294, in _shutdown
    t.join()
  File "/usr/lib/python3.6/threading.py", line 1056, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.6/threading.py", line 1072, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt

Seems like this is the similar discussion.

I would appreciate any tips and helps.

Thank you very much,