How to reclaim the memory in the relax function output, so that it would not oom?

Development unity
#1

Hi,

I current use PackedFunc directly to get the model infered result like “xx=vm->GetFunction(“yy”); out = xx()”. Things goes well, except the output would continuously alloc new memory, which at last cause oom.

I think the logic here is that I shall relaim the memory inside out explictly by myself, but I haven’t find any useful API to server this purpose. So anyone knows how to reclaim the output memory properly?

[UPDATE] The full code is as:

ObjectRef ret = func(xx)
NDArray out = Downcast<NDArray>(ret);
DLManagedTensor *tensor = out.ToDLPack();

Without ToDLPack conversion, it seems to me that it works fine. But with ToDLPack, it would increase storage obj ref cnt, so that it could not be reclaimed. So for this case, how to properly dec the storage obj again?..

Thx~

#2 
{
  ObjectRef ret = func(xx)
  NDArray out = Downcast<NDArray>(ret);
}
// out get de-allocated here

no need to convert it to anything. As long as ret goes out of scope, the memory will get reclaimed. One catch though is that the vm allocates output from memory pool, and current allocator recycles by size. That could mean the output come with different size, right now the memory pool might try to allocate new memory of different size bucket.

1 Like
#3

Yep, only NDArray works fine, and data ptr could be extract with out->data here.