[AutoTVM] Auto-tuning a convolutional network Error

Questions
1

I’m taking a look at AutoTVM, specifically the Auto-tuning a convolutional network tutorial.

I would like to autotune a network on my current machines cuda device. Do you have to use RPC to do this?

I’ve set up RPC, and I’m able to register with the rpc tracker. I’ve changed the target to cuda, and the device key.

I’m never able to fully execute the tutorial.
XGBTuner and cuda target gives me: ValueError: feature_names mismatch: Followed by a long list of feature names

Using the default XGBTuner and llvm target I get the following error:

Extract tasks...
[17:41:54] /tvm/src/runtime/opencl/opencl_device_api.cc:255: Initialize OpenCL platform 'NVIDIA CUDA'
[17:41:54] /tvm/src/runtime/opencl/opencl_device_api.cc:276: opencl(0)='GeForce GTX TITAN' cl_device_id=0x5f7c330
Tuning...
[Task  1/24]  Current/Best:   41.72/  41.72 GFLOPS | Progress: (1/1000) | 0.57 s Done.
Traceback (most recent call last):
  File "tune_nnvm_arm.py", line 337, in <module>
    tune_and_evaluate()
  File "tune_nnvm_arm.py", line 293, in tune_and_evaluate
    tune_tasks(tasks, **tuning_option)
  File "tune_nnvm_arm.py", line 265, in tune_tasks
    tuner_obj.load_history(autotvm.record.load_from_file(tmp_log_file))
  File "/tvm/python/tvm/autotvm/tuner/model_based_tuner.py", line 283, in load_history
    maximums = self.model_optimizer.find_maximums(base_model, self.plan_size, self.visited)
  File "/tvm/python/tvm/autotvm/tuner/sa_model_optimizer.py", line 86, in find_maximums
    new_points[i] = random_walk(p, self.dims)
  File "/tvm/python/tvm/autotvm/tuner/sa_model_optimizer.py", line 145, in random_walk
    from_i = np.random.randint(len(old))
  File "mtrand.pyx", line 993, in mtrand.RandomState.randint
ValueError: low >= high

GATuner and cuda target:

Extract tasks...
[17:52:01] /tvm/src/runtime/opencl/opencl_device_api.cc:255: Initialize OpenCL platform 'NVIDIA CUDA'
[17:52:01] /tvm/src/runtime/opencl/opencl_device_api.cc:276: opencl(0)='GeForce GTX TITAN' cl_device_id=0x5ed4bc0
Tuning...
Traceback (most recent call last):
  File "tune_nnvm_arm.py", line 337, in <module>
    tune_and_evaluate()
  File "tune_nnvm_arm.py", line 293, in tune_and_evaluate
    tune_tasks(tasks, **tuning_option)
  File "tune_nnvm_arm.py", line 273, in tune_tasks
    autotvm.callback.log_to_file(tmp_log_file)])
  File "/tvm/python/tvm/autotvm/tuner/tuner.py", line 125, in tune
    self.update(inputs, results)
  File "/tvm/python/tvm/autotvm/tuner/ga_tuner.py", line 81, in update
    elite_indexes = np.argpartition(scores, -self.elite_num)[-self.elite_num:]
  File "/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 731, in argpartition
    return _wrapfunc(a, 'argpartition', kth, axis=axis, kind=kind, order=order)
  File "/usr/local/lib/python2.7/dist-packages/numpy/core/fromnumeric.py", line 52, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
ValueError: kth(=-2) out of bounds (1)
 Done.

RandomTuner or GridSearchTuner with cuda target:

[Task 24/24]  Current/Best:  410.72/ 410.72 GFLOPS | Progress: (1/1000) | 0.90 s Done.
Compile...
Upload...
Traceback (most recent call last):
  File "tune_nnvm_arm.py", line 337, in <module>
    tune_and_evaluate()
  File "tune_nnvm_arm.py", line 315, in tune_and_evaluate
    remote = autotvm.measure.request_remote(device_key, timeout=10000)
  File "/tvm/python/tvm/autotvm/measure/measure_methods.py", line 60, in request_remote
    host = os.environ['TVM_TRACKER_HOST']
  File "/usr/lib/python2.7/UserDict.py", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'TVM_TRACKER_HOST'
2

Currently we cannot tune for cuda backend. I will send the PR this week. I will let you know when it is ready

3

Thank you!!

I’m still getting the same errors when I change the target to llvm.

I have tried specifically setting the target using the value given from gcc -v and just using llvm

4

Are you referring to the TVM_TRACKER_HOST error? Currently we require setting up an RPC Tracker and passing the address and port via environment variables to the tuning script via TVM_TRACKER_HOST and TVM_TRACKER_PORT.

5

Following the tutorial, I set up an RPC tracker, following the steps given in the tutorial. Are these steps different if I’m tracking tuning on the same machine?

If I run python -m tvm.exec.query_rpc_tracker --host=0.0.0.0 --port=9190 I get the following output:

Tracker address 0.0.0.0:9190

Server List
----------------------------
server-address  key
----------------------------
127.0.0.1:60574 server:this_machine
----------------------------

Queue Status
----------------------------
key             free    pending
----------------------------
this_machine    1       0
----------------------------

It looks like the tracker was set up correctly. Am I missing something?

6

You should use “llvm -device=arm_cpu”. For the last error, I fixed it. Please check out the new tutorial on tvm.ai

7

So does this currently only work for an arm cpu? And cuda in the future?

8

cuda is supported by this