Hexagon copyfrom error

jaeyoon · October 4, 2022, 6:48am

Hi, I am trying to upload one conv layer into hexagon 780 (snapdragon 888)

When the input array is small, (e.x. 64 x 64) it works well

However, it raises an error during copying data into hexagon when the input array is large (like HD image)

I copied 540x540x4 array as follows

xt = tvm.nd.empty(X.shape, X.dtype, sess.device)
xt.copyfrom(np.ones((540,540,4), dtype='int16') # tried float16, ending up with the same error

The returned command output is :

/tvm/python/tvm/runtime/ndarray.py:183: in copyfrom
    check_call(_LIB.TVMArrayCopyFromBytes(self.handle, data, nbytes))
--------------------------------------------------------------------------------
E           tvm.error.RPCError: Traceback (most recent call last):
E             126: 0xffffffffffffffff
E             125: _start
E             124: __libc_start_main
E             123: Py_BytesMain
E             122: Py_RunMain
E             121: PyRun_SimpleFileExFlags
E             120: 0x000000000067fa70
E             119: 0x000000000067f9ce
E             118: 0x000000000067f950
E             117: PyEval_EvalCode
E             116: _PyEval_EvalCodeWithName
E             115: _PyEval_EvalFrameDefault
E             114: _PyFunction_Vectorcall
E             113: _PyEval_EvalCodeWithName
E             112: _PyEval_EvalFrameDefault
E             111: _PyObject_MakeTpCall
E             110: 0x000000000059d89e
E             109: _PyFunction_Vectorcall
E             108: _PyEval_EvalCodeWithName
E             107: _PyEval_EvalFrameDefault
E             106: 0x000000000050bb2d
E             105: _PyEval_EvalFrameDefault
E             104: _PyFunction_Vectorcall
E             103: _PyEval_EvalCodeWithName
E             102: _PyEval_EvalFrameDefault
E             101: PyObject_Call
....
E             14: _PyEval_EvalFrameDefault
E             13: _PyObject_MakeTpCall
E             12: 0x00007fa7b01379eb
E             11: _ctypes_callproc
E             10: 0x00007fa7aff0a409
E             9: 0x00007fa7aff0aff4
E             8: TVMArrayCopyFromBytes
E             7: tvm::runtime::ArrayCopyFromBytes(DLTensor*, void const*, unsigned long)
E             6: tvm::runtime::RPCDeviceAPI::CopyDataFromTo(DLTensor*, DLTensor*, void*)
E             5: tvm::runtime::RPCClientSession::CopyToRemote(void*, DLTensor*, unsigned long)
E             4: tvm::runtime::RPCEndpoint::CopyToRemote(void*, DLTensor*, unsigned long)
E             3: tvm::runtime::RPCEndpoint::HandleUntilReturnEvent(bool, std::function<void (tvm::runtime::TVMArgs)>)
E             2: tvm::runtime::RPCEndpoint::EventHandler::HandleNextEvent(bool, bool, std::function<void (tvm::runtime::TVMArgs)>)
E             1: tvm::runtime::RPCEndpoint::EventHandler::HandleProcessPacket(std::function<void (tvm::runtime::TVMArgs)>)
E             0: tvm::runtime::RPCEndpoint::EventHandler::HandleReturn(tvm::runtime::RPCCode, std::function<void (tvm::runtime::TVMArgs)>)

E             File "/tvm/src/runtime/rpc/rpc_endpoint.cc", line 376
E           RPCError: Error caught from RPC call:
E           [15:42:18] /tvm/src/runtime/hexagon/rpc/android/session.cc:61:
E           ---------------------------------------------------------------
E           An error occurred during the execution of TVM.
E           For more information, please see: https://tvm.apache.org/docs/errors.html
E           ---------------------------------------------------------------
E             Check failed: (rc == AEE_SUCCESS) is false: hexagon_rpc_send failed: -2147482623

/tvm/python/tvm/_ffi/base.py:348: RPCError

Thanks in advance

masahi · October 4, 2022, 6:53pm

Have you updated your TVM recently? I also saw an error when trying to allocate a large buffer, but it should have been fixed by rpc_receive_buffer_size_bytes below bigger.

github.com

apache/tvm/blob/38f53e8c95d6b4387510e38da89b02edb913e886/python/tvm/contrib/hexagon/session.py#L61


    The stack size of the remote device, to be passed to
    tvm.contrib.hexagon.create_hexagon_session.
"""


def __init__(
    self,
    launcher: "HexagonLauncherRPC",
    remote_kw: dict,
    session_name: str = "hexagon-rpc",
    remote_stack_size_bytes: int = 256 * 1024,  # Min size for main thread in QuRT/sim
    rpc_receive_buffer_size_bytes: int = 256 * 1024 * 1024,  # Size for passing hexagon tests
):
    self._launcher = launcher
    self._session_name: str = session_name
    self._remote_stack_size_bytes: int = remote_stack_size_bytes
    self._rpc_receive_buffer_size_bytes: int = rpc_receive_buffer_size_bytes
    self._remote_kw: dict = remote_kw
    self._rpc = None
    self._requires_cpu_device = False
    self._device = None

Previously rpc_receive_buffer_size_bytes was 5 * 1024 * 1024, we changed that about 2 weeks ago.

jaeyoon · October 6, 2022, 7:11am

Thanks @masahi

Then which version of TVM should I check out?

38f53e8c95 commit as you mentioned or some other??

Thanks in advance Jaeyoon

jaeyoon · October 6, 2022, 2:22pm

Recent main branch works well now, Thanks!