[bundle_deploy] Enable multi-threading with bundle_deploy example

keneoneth · June 22, 2024, 6:53am

If I understand correctly, if I compile and run a static case using the same approach as shown in https://github.com/apache/tvm/tree/main/apps/bundle_deploy, the case won’t run with multi-threading parallelism support even though its OPs are scheduled in parallel in TVM.

The reason I believe is that the TVMBackendParallelLaunch() function doesn’t include a thread pool support:

github.com

apache/tvm/blob/main/src/runtime/crt/common/crt_backend_api.c#L55-L60


int TVMBackendParallelLaunch(FTVMParallelLambda flambda, void* cdata, int num_task) {
  TVMParallelGroupEnv env;
  env.num_task = 1;
  flambda(0, &env, cdata);
  return 0;
}

If I would like to make multi-threading available, is it correct to make reference to how TVMBackendParallelLaunch() is supported in this file https://github.com/apache/tvm/blob/36b9535ff364c484d04b384555106731049f44cd/src/runtime/thread_pool.cc#L501-519 ?

Any help is well appreciated