RPC error for large arrays

Hi all, When I try to create a big array using the RPC context, the connection gets reset and the error on the server is: free(): invalid next size (normal)

The minimal reproduction code for this is:

remote = autotvm.measure.request_remote(device_key, device_tracker, device_port, timeout=10000)
ctx = remote.cpu()
a = tvm.nd.array(np.ones((5041,720)).astype('float32'), ctx)
b = tvm.nd.array(np.ones((720,192)).astype('float32'), ctx)

I got this as soon as I updated to the latest master. I am trying to go back in the code base history to find the commit that caused the issue. Any idea about what could that be?

Also, can this be related to: Error while loading params to target in RPC session ?

Thanks, Giuseppe

Hey I’m interested in this problem too, but your example works fine for me. I’ve tested it on a Raspberry Pi 4.

Hi @comaniac,

I investigated the issue, and got the commit that breaks for me: https://github.com/apache/incubator-tvm/commit/afcf9397b60ae7ccf46601cf29828992ca9d5f57

Strange you didn’t reproduce it. Out of curiosity what OS are you running from?

Other thing is the device: I am running the runtime on an aarch64 device (while the pi4 is a aarch32 device). Can that make a difference?

I will open an issue and link to the PR.

Thanks,

Giuseppe

I didn’t realize why that PR will break your case at the first glance, but I’ll wait for your PR to see more details.

I’m using Ubuntu 18.04 on host and Raspbian 10 on Rasp4.

Hi @comaniac

There was a problem with how the ring buffer was being shrunk. This is the PR that fixes the issue: https://github.com/apache/incubator-tvm/pull/5516.

Thanks, Giuseppe

Yeah I saw that PR. Thanks for the reminding.