Do TVM runtime threads occupy CPU persistently? How to sleep them in time?

Questions
#1

I have deployed some models in RK 3288, which completed in for example 5ms. But I found that the TVM threads seems consume CPU continuously after the model inference completed. I tested by the following code:

tvm::runtime::Module mod = (*tvm::runtime::Registry::Get("tvm.graph_runtime.create"))(json_data, mod_syslib, device_type, device_id);
tvm::runtime::PackedFunc set_input = mod.GetFunction("set_input");
set_input("data", data);
tvm::runtime::PackedFunc load_params = mod.GetFunction("load_params");
load_params(params);
tvm::runtime::PackedFunc run = mod.GetFunction("run");
while (True)
{
    run();
    usleep(100000); // in microsecond
}

I used 4 tvm threads. The inference run() will be continued 5ms, then sleeped 100ms. But I used top cmd and saw that the cpu occupation is about 75%, use top -H saw that there were 3 threads occupied cpu core continuesly (RK3288 has 4 arm core in total).

image

My question is how to sleep the threads in time without release the tvm::runtime::Module mod.

1 Like
How to reduce the CPU utilization?
#2

I read the code runtime/thread_pool.cc, found that enviroment var TVM_THREAD_POOL_SPIN_COUNT control the number of iterations to spin before thread sleep, it’s really work in my tests.

#3

How do you set the number of CPUs in the model runtime？

1 Like