Do I need to release module if I use c++ runtime api?

1

I notice that we need to use this line to get a Module from seriailized model for inference:

 Module mod = (*tvm::runtime::Registry::Get("tvm.graph_runtime.create"))(
            sjson, mod_dylib, device_type, device_id);
auto set_input = mod.GetFunction("set_input");

This would returns a pointer, thus I wonder if I need to explicitly free it module in my code, or would tvmruntime free that for me so that I do not need to take specially care about the resources ?

2

No explicit free is necessary. In the C++ land, resources will get automatically freed when all the ref to the object get out of the scope.

3

What about the buffers and functions ?

     TVMArrayAlloc(&in_shape[0], in_ndim, dtype_code, dtype_bits, dtype_lanes, device_type, device_id, &inbuf);
auto set_input = mod.GetFunction("set_input");

Do we not need to explicitly free them in the code either ?

4

If the data comes from the C API(in this case TVMArrayAlloc) then you need to free them(by calling TVMArrayFree). Note that you can also directly use the C++ API(NDArray::Empty) and pass it to the same function, in this case the array will be freed automatically

5

Thanks for replying!!!

How could I copy data to NDArray ? I saw in the tvm/runtime/ndarray.h that there are CopyFrom method but only supports DLTensor or NDArray. If I read the image with opencv(which I can access its data with Mat::data as binary bytes), how could I fill the empty NDArray with the image data please ?

6

We can create a DLTensor inplace(which points to the array) and copy to the NDArray content. But you bring up a good point. Perhaps we should add CopyFromBytes and CopyToBytes as member functions of NDArray.

as we did for the C API https://github.com/apache/incubator-tvm/blob/master/src/runtime/ndarray.cc#L290

A contribution is more than welcomed

7

For CPU based NDArray, currently I do it like this:

std::vector<float> _buffer;

/* ... fill _buffer with your data ... */

_input = NDArray::Empty(input_shape, DataType::Float(32), { kDLCPU, 0 });

memcpy(_input->data, _buffer.data(), shape_size * sizeof(float));

Where shape_size is the result from:

size_t get_shape_size(const std::vector<int64_t>& shape)
{
     return std::accumulate(shape.begin(), shape.end(), int64_t(1), std::multiplies<>());
}

@tqchen A CopyFromBytes and CopyToBytes would be very welcome! I would also like to suggest an “NDArray::FromBytes(...)” that would create an CPU ctx NDArray and use the supplied buffer w/o copying.

8

Glad this convo came up as it made me actually look at what I was doing.

I came up with this quick method to copy bytes to NDArray of any device context. Not sure if there’s a better way:

void CopyFromBytes(NDArray* dest, void* data, size_t nbytes)
{
    TVMContext cpu_ctx;
    cpu_ctx.device_type = kDLCPU;
    cpu_ctx.device_id = 0;
    DLTensor input = *dest->operator->();
    input.ctx = cpu_ctx;
    input.data = data;
    dest->CopyFrom(&input);
}

void CopyToBytes(const NDArray* src, void* data)
{
     TVMContext cpu_ctx;
     cpu_ctx.device_type = kDLCPU;
     cpu_ctx.device_id = 0;
     DLTensor output = *src->operator->();
     output.ctx = cpu_ctx;
     output.data = data;
     src->CopyTo(&output);
}