High threadpool overhead on Windows

1

cc @yidawang @yzhliu @tqchen

Hi, I am having a problem regarding thread pool performance on Windows. My machine is a dual socket Xeon x5650 (12 core 12 Thread).

This PR by @yidawang is supposed to improve multi-threading performance. But I found that although that PR brought CPU usage to 100%, the overall inference time didn’t get faster than the original thread pool. The CPU usage with the original thread pool is around 75%.

Here is the VTune result with the original thread pool. The CPU utilization is not ideal, but there seems to be no overhead from multi threading.

image
image1280×720 197 KB

Here is the VTune result with the thread pool from the latest TVM. Although the CPU util becomes much better, the elapsed time is almost identical with the result above. VTune is reporting high multi-threading overhead.

image
image1280×720 192 KB

image
image1280×720 617 KB

I have following questions:

  1. Did the PR by @yidawang actually improve end to end performance on high core count Linux machines?
  2. Does similar multi-threading overhead exist on high core count Linux machines?
  3. Can we replace TVM’s thread pool implementation with Microsoft’s concurrency::parallel_for(), which is readily available on Windows ?
2

In our experiment, that PR does improve performance on EC2 c5/c4 instances (up to 9xlarge), with Linux (ubuntu) installed, while with c5.18xlarge (36core), it does not perform better than 9xlarge - maybe because of the effect of NUMA.

I’m not familiar with Microsoft’s concurrency::parallel_for()

3

Thanks. Microsoft’s concurrency::parallel_for() works similarly with OpenMP’s parallel pragma and Intel’s tbb::parallel_for(). It takes a loop range and a lambda function to run in parallel.