AutoScheduler failed to find a valid schedule

[gbxu]

thanks a lot @comaniac ! I read the rpc related issues then located my bug.

Before, I tried to parallelize my searching using multiple scripts.

CUDA_VISIBLE_DEVICES=0 python3 my_tvm_searching.py ...
CUDA_VISIBLE_DEVICES=1 python3 my_tvm_searching.py ...
...

my_tvm_searching.py will launch the localrpc, and it’s not the correct usage.

    measure_ctx = auto_scheduler.LocalRPCMeasureContext(min_repeat_ms=300)
    tune_option = auto_scheduler.TuningOptions(
        num_measure_trials=100,
        runner=measure_ctx.runner,
        measure_callbacks=[auto_scheduler.RecordToFile(log_file)],
        builder=tvm.auto_scheduler.LocalBuilder(timeout=100),
        verbose=2,
    )
    task.tune(tune_option)
    del measure_ctx

after fixing bug, my commands are as follow:

nohup python3 -m tvm.exec.rpc_tracker --host=0.0.0.0 --port=9190 & \
CUDA_VISIBLE_DEVICES=0 nohup python3 -m tvm.exec.rpc_server --tracker 127.0.0.1:9190 --key V100 --host 0.0.0.0 --port=9091 & \
CUDA_VISIBLE_DEVICES=1 nohup python3 -m tvm.exec.rpc_server --tracker 127.0.0.1:9190 --key V100 --host 0.0.0.0 --port=9092 & 
...
python3 my_tvm_searching.py

my_tvm_searching.py:

...
        runner = tvm.auto_scheduler.RPCRunner(key="V100", host="localhost", port=9190, n_parallel=8, min_repeat_ms=300, timeout=1000)
    tune_option = auto_scheduler.TuningOptions(
        num_measure_trials=100,  # change this to 1000 to achieve the best performance
        runner=runner,
        measure_callbacks=[auto_scheduler.RecordToFile(log_file)],
        builder=tvm.auto_scheduler.LocalBuilder(timeout=1000),
        verbose=2,
    )
    task.tune(tune_option)
...

NOTE: set n_parallel as #gpus, set timeout a proper number.

ref: https://tvm.apache.org/docs/how_to/tune_with_autoscheduler/tune_network_arm.html?highlight=parallel https://tvm.apache.org/docs/reference/api/python/auto_scheduler.html?highlight=tvm%20auto_scheduler%20rpcrunner#tvm.auto_scheduler.RPCRunner

[AutoTuning] Specifying autotvm runner for local GPUs on the same machine

Hi @merrymercy Thank you for your reply! Yes, I’ve tried to use the environment variable to register the GPU device to the rpc server, as follow: CUDA_VISIBLE_DEVICES=0 python3 -m tvm.exec.rpc_server --key titanv100 --tracker=0.0.0.0:9190 However, I’m seeing an error that seems to related to running the generated library through rpc on this GPU. The error happens in the run_through_rpc() function in tvm/python/tvm/autotvm/measure/measure_methods.py When doing autotuning on GPU, AutoTVM seem…

How to tune CNN networks with multiple gpu devices?

Thanks for your reply. I just tried this method recently but got a few more problems now. I launched the auto-tune script in one of my computing server with 8 titan-v gpus. I found that all the cpu cores are utilized while the gpu devices are almost not used at all during the entire tuning process. I launched the auto-tune script in my master node (i.e. server A). There are 3 other servers registered as computing node (python -m tvm.exec.rpc_server --tracker=${HOST_IP}:9190 --key=titanv). …

[AutoTuning] Specifying autotvm runner for local GPUs on the same machine

Hi @merrymercy Thank you for your reply! Yes, I’ve tried to use the environment variable to register the GPU device to the rpc server, as follow: CUDA_VISIBLE_DEVICES=0 python3 -m tvm.exec.rpc_server --key titanv100 --tracker=0.0.0.0:9190 However, I’m seeing an error that seems to related to running the generated library through rpc on this GPU. The error happens in the run_through_rpc() function in tvm/python/tvm/autotvm/measure/measure_methods.py When doing autotuning on GPU, AutoTVM seem…

Autoscheduling Multiple Shapes Sequentially Results In Unexpected Measure Timeouts

Hi everyone, Im hoping for some guidance on what I might be doing wrong. What im trying to do is tune multiple operators/shapes sequentially in a script using auto_scheduler. My code looks roughly as follows: for shape in shapes_list: tune_option = auto_scheduler.TuningOptions( num_measure_trials=num_trials, num_measures_per_round=num_measures_per_round, verbose=2, measure_callbacks=[ auto_scheduler.RecordToFile( …

How to tune CNN networks with multiple gpu devices?

Hi, I noticed that the context is always set to gpu 0 in this tutorial like below: ctx = tvm.context(str(target), 0) How can I use all gpus availiable to accelerate the tuning process?