[RFC] Unified Static Memory Planning

Hi @areusch ,

That broadly aligns with our thinking.

The proposed USMP’s actual “component” interface will be quite similiar to TVMC CLI additions. Therefore, graph executor flow could use “–with-parameter-buffer” to make USMP expose the parameter buffer to the actual executor runtime – so that the executor could update constants with known offsets.

Regarding specific parameter updates, since the relay pipeline run passes such FoldConstants, would it be safe to do specific parameter updates ? Anyway, if thats the case, we could use the same way it uses to know which parameters to update to using offsets instead.

Ack, yes maybe we should not limit this in the design.

I see, I think you are querying about attributes of the pools itself.

Initially, we are starting with “name” and “target” to identify the pool uniquely and which targets could access them, respectively.

However, going forward we are going to provide a guide “size” for the buffer to be used, which we could use to distribute tensors (if there are options) based on memory pressure – hence the guide.

Going a bit further out, we are planning append more metadata such as “bandwith” for the buffers, but to be used by scheduler to redact pools based on where they want them placed (in some cases where we might use double_buffering, rolling_buffers using scheduling primitives) – that goes hand-in-hand with performance required.