RPCError: RPCTimeEvaluator expects more arguments

Hi, i am trying to tune resnet18_v2 on a Jetson AGX Orin. I have been able to set up the RPC connection and execute the model successfully on the device. Tuning on the other hand does not work. Apparently, the uploaded modules are not executed.

I get this output on my host machine: [Task 16/16] Current/Best: 0.00/ 0.00 GFLOPS | Progress: (32/32) | 17.16 sWARNING:root:Could not find any valid schedule for task Task(func_name=conv2d_nchw.cuda, args=((‘TENSOR’, (1, 3, 224, 224), ‘float32’), (‘TENSOR’, (64, 3, 7, 7), ‘float32’), (2, 2), (3, 3, 3, 3), (1, 1), ‘float32’), kwargs={}, workload=(‘conv2d_nchw.cuda’, (‘TENSOR’, (1, 3, 224, 224), ‘float32’), (‘TENSOR’, (64, 3, 7, 7), ‘float32’), (2, 2), (3, 3, 3, 3), (1, 1), ‘float32’)). A file containing the errors has been written to /tmp/tvm_tuning_errors_l46epbfb.log

The server outputs many messages like this: 2023-10-23 20:21:19.121 INFO start serving at /tmp/tmpmqhjx_x9 2023-10-23 20:21:19.232 INFO load_module /tmp/tmpmqhjx_x9/tmp_func_c6c358d5cbc73487.tar 2023-10-23 20:21:19.628 INFO finish serving (‘127.0.0.1’, 41758)

The log file from above contains the following error messages: cat /tmp/tvm_tuning_errors_d5prl_x1.log Traceback (most recent call last): … RPCError: Error caught from RPC call: [19:32:58] /opt/tvm/include/tvm/runtime/packed_func.h:1814: Function runtime.RPCTimeEvaluator(0: runtime.Module, 1: basic_string, 2: int, 3: int, 4: int, 5: int, 6: int, 7: int, 8: int, 9: int, 10: int, 11: basic_string) → runtime.PackedFunc expects 12 arguments, but 11 were provided.

I don’t understand, why RPCTimeEvaluator would expect 12 arguments. In its signature in rpc_module.cc it looks like it only expects 11 args.

Does anyone know, whats going on? Am I missing something here?

In version v0.13.0 the runtime.RPCTimeEvaluator function didn’t have the 12th cache_flush_bytes parameter but it does in version v0.14.0 with this PR, so perhaps the Jetson doesn’t have the same version of the TVM runtime.

I think you can check tvm._ffi.libinfo.__version__ on both machines.

1 Like

Thank you, that helped!