[MultiThread][ThreadPool] Performance degradation when running relay module in multiple threads

Questions
1

  • Device: Skylake 8163 with 48 pysical cores

  • Env Setting: TVM_BIND_THREADS=0 TVM_NUM_THREADS=4

  • Code Snippet:

module = graph_executor.GraphModule(lib["default"](ctx))
def thread_run:
   for i in range(repeats):
       module.run()
threads = []
   for i in range(num_threads):
      threads.append(PropagatingThread(
           target=process_run,
      ))

Since there are still 4 physical cores occupied by each thread in 2-threaded run, the performace is expected to be the same as single-threaded run. But the performace of each thread in 2-threaded run actually is only as 50% of single-threaded run, and 4 -threaded run is only as 25% and so on…

Any idea about the performance degradation in multi-threaded run?

1 Like
2

Help to at if anyone have experience on this

@tqchen @junrushao @comaniac @FrozenGene :smiley:

3

How about setting one environment TVM_THREAD_POOL_SPIN_COUNT = 0 , does it improve your case?

4

Thanks. It doesn’t improve. Performace in both single-threaded run and multi-threaded run drop 10-20%, and CPU utilization drops from nearly 100% to nearly 50% with TVM_THREAD_POOL_SPIN_COUNT=0, which I think is reasonable.

5

Add an addtional info: in multi-processed run, the performace of module.run() in each process does not drop.

6

if we change the thread pool to OpenMP, do we have the same problem?

7

Yes, it also has the same problem running on omp as threading backend. We didn’t see much difference between TVM ThreadPool and OMP in such cases.

8

Do you mean if we use Python’s multi process, you could get ideal result but use Python’s multi thread, you get bad result? Or what else things you mean?

9

By “multi-prcocess” I mean run the python script two times simutaniously, while “multi-thread” is implemented by threading.Thread in python script.

10

I found out that it might the problem of python thread.Thread instead of TVM.

11

Threads in python aren’t actually executed concurrently due to the GIL. So the reason everything is slower is because you aren’t actually doing anything in parallel. Also, it could be a lot of you time is actually spent in the python interpreter instead of executing you model. You should try using time evaluator (tvm.runtime — tvm 0.8.dev0 documentation) instead of your own python loop.

2 Likes
12

As @tkonolige mentioned, never ever assume python’s multi-threading could help in most of the cases >_<

13

Got it, thanks! @tkonolige @junrushao