masahi
September 16, 2020, 8:34pm
1
Hi, what is the correct way to use time evaluator with VM runtime? Or is it supported at all? I quickly tried the following
vm_exec = relay.vm.compile(mod, target=target, params=params)
vm = VirtualMachine(vm_exec, ctx)
ftimer = vm.module.time_evaluator("run", ctx, number=1, repeat=50)
but got the error
tvm._ffi.base.TVMError: Traceback (most recent call last):
[bt] (4) /mnt/2e797a66-fd2b-44fc-a3ba-24d7d65f2780/projects/dev/tvm/build/libtvm.so(TVMFuncCall+0x69) [0x7f1b88d013f9]
[bt] (3) /mnt/2e797a66-fd2b-44fc-a3ba-24d7d65f2780/projects/dev/tvm/build/libtvm.so(+0x11e0302) [0x7f1b88d6b302]
[bt] (2) /mnt/2e797a66-fd2b-44fc-a3ba-24d7d65f2780/projects/dev/tvm/build/libtvm.so(tvm::runtime::ModuleNode::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+0x4d) [0x7f1b88d1efcd]
[bt] (1) /mnt/2e797a66-fd2b-44fc-a3ba-24d7d65f2780/projects/dev/tvm/build/libtvm.so(tvm::runtime::vm::VirtualMachine::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)+0x279) [0x7f1b88d4d169]
[bt] (0) /mnt/2e797a66-fd2b-44fc-a3ba-24d7d65f2780/projects/dev/tvm/build/libtvm.so(+0x11bd5bb) [0x7f1b88d485bb]
File "/home/masa/projects/dev/tvm/src/runtime/vm/vm.cc", line 173
TVMError: Unknown packed function: run
cc @zhiics @kevinthesun
vm.module.time_evaluator("invoke", ctx, number=1, repeat=10)
masahi
September 16, 2020, 8:52pm
3
ok, how do you run the timer? It looks like I need to pass arguments. If I do
ftimer = vm.module.time_evaluator("invoke", ctx, number=1, repeat=50)
prof_res = np.array(ftimer().results) * 1e3
I got
TVMError: Check failed: i < num_args (0 vs. 0) : not enough argument passed, 0 passed but request arg[0].
If I pass a numpy array, I got
prof_res = np.array(ftimer(data.numpy()).results) * 1e3
File "/home/masa/projects/dev/tvm/python/tvm/runtime/module.py", line 225, in evaluator
blob = feval(*args)
File "/home/masa/projects/dev/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 223, in __call__
values, tcodes, num_args = _make_tvm_args(args, temp_args)
File "/home/masa/projects/dev/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 188, in _make_tvm_args
raise TypeError("Don't know how to handle type %s" % type(arg))
TypeError: Don't know how to handle type <class 'numpy.ndarray'>
If I pass TVM ndarray, I got
TVMError: Check failed: type_code_ == kTVMStr (13 vs. 11) : expected str but get NDArrayContainer
zhiics
September 16, 2020, 9:50pm
4
I think you can initialize the VM and set up inputs first first (or use run with args). Then you can do:
vm.module.time_evaluator("invoke", ctx, number=1, repeat=10)
here is an example:
def get_vm_output(mod, data, params, target, ctx, dtype="float32", number=2, repeat=20):
with tvm.transform.PassContext(opt_level=3):
exe = vm.compile(mod, target, params=params)
rly_vm = vm_rt.VirtualMachine(exe, ctx)
result = rly_vm.run(data)
if measure:
print("Evaluate vm inference cost of {} on {}".format(model, repr(ctx)))
ftimer = rly_vm.module.time_evaluator("invoke", ctx, number=number, repeat=repeat)
# Measure in millisecond.
prof_res = np.array(ftimer("main", data).results) * 1000
print(
"Mean vm inference time (std dev): %.2f ms (%.2f ms)"
% (np.mean(prof_res), np.std(prof_res))
)
return result.asnumpy().astype(dtype)
1 Like
masahi
September 16, 2020, 11:33pm
5
Thanks, good to know there is an example. The following seems to work. Passing a numpy array won’t work, need to wrap with tvm ndarray.
ftimer("main", tvm.nd.array(np_arr))
The following also works, if the argument is set beforehand
ftimer("main")