Time evaluator for VM

masahi · September 16, 2020, 8:34pm

Hi, what is the correct way to use time evaluator with VM runtime? Or is it supported at all? I quickly tried the following

    vm_exec = relay.vm.compile(mod, target=target, params=params)
    vm = VirtualMachine(vm_exec, ctx)
    ftimer = vm.module.time_evaluator("run", ctx, number=1, repeat=50)

but got the error

tvm._ffi.base.TVMError: Traceback (most recent call last):
  [bt] (4) /mnt/2e797a66-fd2b-44fc-a3ba-24d7d65f2780/projects/dev/tvm/build/libtvm.so(TVMFuncCall+0x69) [0x7f1b88d013f9]
  [bt] (3) /mnt/2e797a66-fd2b-44fc-a3ba-24d7d65f2780/projects/dev/tvm/build/libtvm.so(+0x11e0302) [0x7f1b88d6b302]
  [bt] (2) /mnt/2e797a66-fd2b-44fc-a3ba-24d7d65f2780/projects/dev/tvm/build/libtvm.so(tvm::runtime::ModuleNode::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)+0x4d) [0x7f1b88d1efcd]
  [bt] (1) /mnt/2e797a66-fd2b-44fc-a3ba-24d7d65f2780/projects/dev/tvm/build/libtvm.so(tvm::runtime::vm::VirtualMachine::GetFunction(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, tvm::runtime::ObjectPtr<tvm::runtime::Object> const&)+0x279) [0x7f1b88d4d169]
  [bt] (0) /mnt/2e797a66-fd2b-44fc-a3ba-24d7d65f2780/projects/dev/tvm/build/libtvm.so(+0x11bd5bb) [0x7f1b88d485bb]
  File "/home/masa/projects/dev/tvm/src/runtime/vm/vm.cc", line 173
TVMError: Unknown packed function: run

cc @zhiics @kevinthesun

kevinthesun · September 16, 2020, 8:44pm

vm.module.time_evaluator("invoke", ctx, number=1, repeat=10)

masahi · September 16, 2020, 8:52pm

ok, how do you run the timer? It looks like I need to pass arguments. If I do

    ftimer = vm.module.time_evaluator("invoke", ctx, number=1, repeat=50)
    prof_res = np.array(ftimer().results) * 1e3

I got

TVMError: Check failed: i < num_args (0 vs. 0) : not enough argument passed, 0 passed but request arg[0].

If I pass a numpy array, I got

    prof_res = np.array(ftimer(data.numpy()).results) * 1e3
  File "/home/masa/projects/dev/tvm/python/tvm/runtime/module.py", line 225, in evaluator
    blob = feval(*args)
  File "/home/masa/projects/dev/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 223, in __call__
    values, tcodes, num_args = _make_tvm_args(args, temp_args)
  File "/home/masa/projects/dev/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 188, in _make_tvm_args
    raise TypeError("Don't know how to handle type %s" % type(arg))
TypeError: Don't know how to handle type <class 'numpy.ndarray'>

If I pass TVM ndarray, I got

TVMError: Check failed: type_code_ == kTVMStr (13 vs. 11) : expected str but get NDArrayContainer

zhiics · September 16, 2020, 9:50pm

I think you can initialize the VM and set up inputs first first (or use run with args). Then you can do:

vm.module.time_evaluator("invoke", ctx, number=1, repeat=10)

here is an example:

github.com

apache/incubator-tvm/blob/34f33431776a383999243aabdadf12db1eab13b4/tests/python/relay/benchmarking/benchmark_vm.py#L63-L79


def get_vm_output(mod, data, params, target, ctx, dtype="float32", number=2, repeat=20):
    with tvm.transform.PassContext(opt_level=3):
        exe = vm.compile(mod, target, params=params)
        rly_vm = vm_rt.VirtualMachine(exe, ctx)
        result = rly_vm.run(data)

    if measure:
        print("Evaluate vm inference cost of {} on {}".format(model, repr(ctx)))
        ftimer = rly_vm.module.time_evaluator("invoke", ctx, number=number, repeat=repeat)
        # Measure in millisecond.
        prof_res = np.array(ftimer("main", data).results) * 1000
        print(
            "Mean vm inference time (std dev): %.2f ms (%.2f ms)"
            % (np.mean(prof_res), np.std(prof_res))
        )

    return result.asnumpy().astype(dtype)

masahi · September 16, 2020, 11:43pm

Thanks, good to know there is an example. The following seems to work. Passing a numpy array won’t work, need to wrap with tvm ndarray.

ftimer("main", tvm.nd.array(np_arr))

The following also works, if the argument is set beforehand

ftimer("main")