Cpp_rpc fail to connect

srkreddy1238 · June 25, 2021, 1:47pm

Latest version results in below exception

python3 -m tvm.exec.rpc_tracker --host=127.0.0.1 --port=9090
INFO:RPCTracker:bind to 127.0.0.1:9091
ERROR:asyncio:Exception in callback BaseAsyncIOLoop._handle_events(10, 1)
handle: <Handle BaseAsyncIOLoop._handle_events(10, 1)>
Traceback (most recent call last):
File "/local/mnt/workspace/sivb/anaconda3/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "/local/mnt/workspace/sivb/anaconda3/lib/python3.8/site-packages/tornado/platform/asyncio.py", 
line 139, in _handle_events
handler_func(fileobj, events)
File "/local/mnt/workspace/sivb/TVM/CI/tvm-bk/python/tvm/rpc/tornado_util.py", line 41, in 
_event_handler
self._event_handler(events)
File "/local/mnt/workspace/sivb/TVM/CI/tvm-bk/python/tvm/rpc/tornado_util.py", line 79, in 
_event_handler
if self._update_read() and (events & self._ioloop.WRITE):
File "/local/mnt/workspace/sivb/TVM/CI/tvm-bk/python/tvm/rpc/tornado_util.py", line 118, in 
_update_read
self.on_message(msg)
File "/local/mnt/workspace/sivb/TVM/CI/tvm-bk/python/tvm/rpc/tracker.py", line 223, in on_message
self.call_handler(json.loads(msg))
File "/local/mnt/workspace/sivb/TVM/CI/tvm-bk/python/tvm/rpc/tracker.py", line 279, in call_handler
if info["addr"][0] is None:
KeyError: 'addr'

A bit of debug pointing to [Bug Fixed] Make query_rpc_tracker show the correct device server por… · apache/tvm@0bbaf0e · GitHub

where the addr is updated for python server but not for cpp_rpc.

@lmxyy can you check this ?

lmxyy · June 25, 2021, 2:12pm

Hi, could you elaborate more on this problem? I’ve never met it before during my usage. I think this is caused by the inconsistent TVM version on the tracker and server side.

srkreddy1238 · June 26, 2021, 6:12am

Pls ignore, It happened on one build. Didn’t happen later, I will get back if I come across again.

srkreddy1238 · June 26, 2021, 6:18am

I tried passing addr attribute when I got above error and it worked fine for me.

github.com

apache/tvm/blob/5177729ce3f20b0073c334403136af47b163bb55/apps/cpp_rpc/rpc_tracker_client.h#L82


void TryConnect() {
  if (!tracker_addr_.empty() && (tracker_sock_.IsClosed())) {
    tracker_sock_ = ConnectWithRetry();


    int code = kRPCTrackerMagic;
    ICHECK_EQ(tracker_sock_.SendAll(&code, sizeof(code)), sizeof(code));
    ICHECK_EQ(tracker_sock_.RecvAll(&code, sizeof(code)), sizeof(code));
    ICHECK_EQ(code, kRPCTrackerMagic) << tracker_addr_.c_str() << " is not RPC Tracker";


    std::ostringstream ss;
    ss << "[" << static_cast<int>(TrackerCode::kUpdateInfo) << ", {\"key\": \"server:" << key_
       << "\"}]";
    tracker_sock_.SendBytes(ss.str());


    // Receive status and validate
    std::string remote_status = tracker_sock_.RecvBytes();
    ICHECK_EQ(std::stoi(remote_status), static_cast<int>(TrackerCode::kSuccess));
  }
}
/*!
 * \brief Close Clean up tracker resources.

Anyway, it’s not happening now.

echuraev · June 28, 2021, 7:54am

I think this problem is the same as I met for Android RPC: [RPC] Fix android rpc connection to tracker by echuraev · Pull Request #8327 · apache/tvm · GitHub This problem was introduced by this patch: [Bug Fixed] Make query_rpc_tracker show the correct device server por… · apache/tvm@0bbaf0e · GitHub