Is Module thread-safe?

Questions
#1

For example, if I do

PackedFunc func = module_ptr_->GetFunction(func_name, false);
func.CallPacked(tvm_args, &rv);

where module_ptr is initialized globally, can I call the above piece of code in parallel?

#2

If you init module_ptr_ inside a c++ function statically, init-only-once is guaranteed since C++ 11 standard. Not sure if there is any internal states of Module.

#3

yup. wondering whether Module itself is thread-safe or not.

1 Like
#4

No, I don’t think it is thread safe. When you use C++ API, you retrieve set_input, run, get_output functions from Module. These functions are obviously tied to the internal state of Module, so they are not supposed to be called concurrently.

When I run multiple inferences in parallel, I create module for each thread.

#5

Thank you for the reply! So does it imply that using a thread_local module could resolve these issues?

Update: i don’t think so…

#6

Are you referring to graph executor module?

#7

By module I mean the return value of tvm.graph_runtime.create(…)

This is a typical usage of C++ API. What I mean is that set_input, run, and get_output below are not thread safe.

	tvm::runtime::Module mod = (*tvm::runtime::Registry::Get("tvm.graph_runtime.create"))(json_data, mod_syslib, device_type, device_id);
	tvm::runtime::PackedFunc load_params = mod.GetFunction("load_params");
	
	tvm::runtime::PackedFunc set_input = mod.GetFunction("set_input");
	tvm::runtime::PackedFunc run = mod.GetFunction("run");
	tvm::runtime::PackedFunc get_output = mod.GetFunction("get_output");
1 Like
#8

Yep, it makes a lot of sense if you are referring to graph runtime module.

I assume Yizhi is more interested in loading a pre-compiled module for tensor expressions, not a full neural net. For example, this.

@yzhliu Did I get it right?

#9

graphruntime are not threadsafe due to set/get function, but the generally the PackedFunc and Module(CUDA/C++) are threadsafe

#10

See related code that locks necessary resrouce to ensure thread safety https://github.com/dmlc/tvm/blob/master/src/runtime/cuda/cuda_module.cc#L108

#11

Module is thread safe, but how to make sure PackedFunc is thread safe? If I get the PackedFunc from one Module and save it to another new thread, there will be segmentation fault when I try to access the PackedFunc in the new thread.

#12

is “GraphRuntime::Run” threadsafe ?

#13

graph runtime functions are not thread safe since they access per runtime state

1 Like
#14

Thank you for your reply