[C++ RPC + AOT] Error "Can not pass in local device" when deploying AOT-compiled model via C++ RPC client

Questions
1

Hi TVM community,

I’m trying to deploy an AOT-compiled graph executor model using the TVM C++ RPC API, but I keep encountering a runtime error related to device handling. Unfortunately, documentation and examples for the C++ RPC client usage are very limited, so I’m seeking guidance.

Environment

  • Platform: Linux (aarch64)
    Linux lima-ubuntu 6.8.0-87-generic #88-Ubuntu SMP PREEMPT_DYNAMIC Sat Oct 11 09:16:38 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux
  • TVM version: v0.12.0 (built from source)
  • RPC server: Running via ./tvm_rpc server --host=0.0.0.0 --port=9000

Model Compilation (Python)

I compiled a simple add(x, y) model in AOT mode with the C++ runtime:

import tvm
from tvm import relay

x = relay.var("x", shape=(1, 3, 224, 224), dtype="float32")
y = relay.var("y", shape=(1, 3, 224, 224), dtype="float32")
z = relay.add(x, y)
func = relay.Function([x, y], z)

mod = tvm.IRModule.from_expr(func)
target = "llvm"

with tvm.transform.PassContext(opt_level=3):
    lib = relay.build(
        mod,
        target=tvm.target.Target(target),
        params={},
        runtime=relay.backend.Runtime("cpp"),
        executor=relay.backend.Executor("aot"),
    )

lib.export_library("aot_mod.so")

Error on C++ Client

When running my C++ client, I get the following error:

Error: [16:34:29] /path/to/tvm/src/runtime/rpc/rpc_module.cc:153:
---------------------------------------------------------------
An error occurred during the execution of TVM.
For more information, please see: https://tvm.apache.org/docs/errors.html
---------------------------------------------------------------
  Check failed: (IsRPCSessionDevice(dev)) is false: Can not pass in local device
Stack trace:
  0: tvm::runtime::RPCWrappedFunc::RemoveSessMask(DLDevice) const
        at /path/to/tvm/src/runtime/rpc/rpc_module.cc:153
  ...
  5: main
        at /path/to/rpc_deploy.cpp:148

C++ Client Code (simplified)

// Connect to RPC server
auto connect_func = tvm::runtime::Registry::Get("rpc.Connect");
auto remote_session = (*connect_func)(server_ip, server_port, "", false);

// Upload and load module on remote
f_upload("/tmp/mymod.so", mod_data);
mod = f_load_module("/tmp/mymod.so");

// This line triggers the error:
DLDevice dev = {kDLCPU, 0};
tvm::runtime::Module gmod = mod.GetFunction("default")(dev);  // ← ERROR HERE

Questions

  1. When using AOT + RPC, should the DLDevice passed to mod["default"] be a remote device?
    But DLDevice is just a struct — how do I specify it belongs to the RPC session?

  2. Is AOT + RPC fully supported in the C++ runtime? Or should I switch to Graph Executor (non-AOT) for RPC deployments?

Any working C++ RPC + AOT example would be extremely helpful!

Thanks in advance for your support!