[microTVM][Memory]Some questions about AllocDataSpaceWithScope

microTVM
#1

Hi, @areusch @ziheng .

When i run tvm.micro.create_local_graph_executor() in python file for deploying a tflite model to stm32f746 board, minirpc will send RPCCode::kDevAllocDataWithScope code to board to run TVMDeviceAllocDataSpaceWithScope function.

Some questions about the function of TVMDeviceAllocDataSpaceWithScope:

  1. What’s the meaning about the ‘Scope’?
  2. What’s this memory(allocate by this function) used for? And when will this part of memory be freed?

Sincerely looking forward to your reply.

#2

@lumos thanks for your question.

Currently, Scope has no effect in the C runtime (you can think of this as just a standard allocate call). At present, scope only serves as a hint to the C++ Device API, and certain (e.g. GPU) implementations use this to distinguish between thread-local vs shared GPU memory allocations. However, when [RFC] Unified Static Memory Planning lands, scope will then refer to a specific memory pool. Then, when [RFC] [uTVM] Embedded C Runtime Interface lands, you’ll specify the base address of these regions through the TVM<model>Memory struct (most of this is in, I believe, but the unification of the two is not).

Currently this is used for scratchpad memory and intermediate tensors. The USMP RFC referenced above will remove the need to use this allocate function for static models.

#3

Hi @areusch , thanks for your reply.

One more question about ‘AllocWithScope’ function:

When tvm.micro.create_local_graph_executor invoke AllocDataSpaceWithScopefunciton to allocate a big memory in with tvm.micro.Session part, I found that this part of memory will NOT be freed after Session exit. if someone run create_local_graph_executor several time without project.flash() or rebooting the board , it would lead to out of memory because AllocDataSpaceWithScope function always allocate memory without releasing memory. So, i wonder if some mistakes in my code, or it’s just a bug in microTVM.

Another question about USMP RFC:

How is the USMP RFC progress? About when will it be completed and merged into main branch?

PS: I’m so excited to use the memory planning mentioned in USMP RFC. I hope it can boost microTVM to deploy more complicate model into resource-constrained device(especially memory-constrained device).

#4

@lumos what happens if you delete the GraphExecutor like so:

graph_executor = tvm.micro.create_local_graph_executor(...)
# run graph
del graph_executor

this should remove the memory.

you can follow the tracking issue for more. many of the tasks listed there have PRs out