cudaDeviceSynchronize should be called after run being called(tvm 0.6)

Troubleshooting
#1

Hi Experts,

I have one face detection application using TVM. And the data flow as following diagram, and all process is in the same thread:

image
image675×636 14.6 KB

In nVidia GPU Environment.

I found the first cudaStreamSynchronize call cost much long time after calling my asynchronous kernel(in Process Output Data). And my kernel aims to process small data. As the image(captured from NIVIDIA Nsight Systems) shows, it costs more than 47ms, but the cost after the first kernel is small:

image
image1438×189 12.9 KB

If i do not call run or call cudaDeviceSynchronize right after calling run, all cudaStreamSynchronize will not cost much time, as the image shows:

1, if do not call run, all cudaStreamSynchronize and kernel run fast:

image
image975×168 9.77 KB

2, if call cudaDeviceSynchronize right after calling run, and cudaDeviceSynchronize cost much time, and all cudaStreamSynchronize and kernel run fast:

image
image1379×168 11.7 KB

Also, i found that no matter how large the input data size is, all the cost time of the run is the same (just more than 1ms and less than 2ms). And i think it is not possible if the run is synchororized.

So, i guess:

synchronize should be called explicitly.

And i really hope any experts can help confirming this.

Thanks in advance!!!

#2

When you call asnumpy to access the data, TVM will synchronize the cuda.

#3

Hi thanks for response first. Actually we use TVM C++ API.

#4

Hi,I am encountering the same trouble.cudaStreamSynchronize cost too much time after call run of tvm,is there any solution?