Multi-tasks on multi-CPU

image

Hi,

I have a question about parallelizing tasks from the graph generated with TVM: Let’s say there are 2 tasks (fused_nn_conv2d_31 and fused_nn_conv2d_3), as in the figure below. Besides we have 2 CPUs, how can I give enough information to TVM so that these 2 tasks will be executed in parallel, 1 per CPU?

Thank you

There is task parallelism and data parallelism. The approach you described in the question - this is task paralellism. TVM does not have it for now, at the same time there is an initiative to introduce task parralelism: [RFC] Compute graph pipeline with new subgraph executor

TVM currently submit whole task for execution and it depends on the underneath solution how to parallel the task inside. For example for GPU it is done quite naturally, GPU by its ideology has SIMD groups and there are multiple threads executed same instructions simultaneously.

Or for CPU TVM can utilize selfmade solution or 3rd parties for doing data parallelism. By default it is used self developed pool of threads or OpenMP can be plugged. Even with default solution you can manage how many threads to use for data parallelism - there are couple environment variable affecting this - TVM_NUM_THREADS or OMP_NUM_THREADS, can be seen here. BTW, it is not necessary that best inference time happens with default value of threads, especially in the application having other compute. You can try to change such env variable to evaluate if you can get better inference time

1 Like