How about setting one environment TVM_THREAD_POOL_SPIN_COUNT = 0 , does it improve your case?
TVM_THREAD_POOL_SPIN_COUNT = 0