Almost 8% tasks have RuntimeDeviceError with AutoScheduler

Troubleshooting
#1

I am running Autoscheduler on Resnet-18 targetting CUDA, with two RPC servers on two GeForce RTX 3090 devices. As much as 8% of measurements seem to be failing with a RuntimeDeviceError. Is this normal and will this limit the search? This is a shared server and other users may be using the GPUs.

all_errs_pct
all_errs_pct1920×964 98.2 KB

#2

RuntimeDeviceError seems to also occur occasionally even if no one else is using the GPUs.