Measure Memory Allocation using Debug Executor

Questions
#1

Hi,

is there a way to measure the peak memory allocation of individual functions using the debug executor?

#2

Right now there is not a way to collect this information. I’d like to add one, but it is a little bit complicated.

1 Like
#3

I think @jwfromm may have a TIR pass for this, but it may be out-of-date.

1 Like
#4

you may be able to use this recently-added analysis pass in combination with graph JSON information to compute the peak memory allocations. you’ll likely need to modify tvm.relay.build_module to call this–it may be tricky to use from pure TVM user code.

1 Like
#5

Hi @max1996 ,

Have a go at this : Improved MLF to contain workspace info by manupa-arm · Pull Request #7938 · apache/tvm · GitHub

The metadata.json is augmented to include peak memory usage in the PR. Please note if you are using crt graph runtime, it might copy the weights (~ (if link-params is used && load_params is called) ) and also maintain a copy of I/O tensors in the workspace. For those reasons, that size is also mentioned in the .json.

Moreover, I’ve added instrumentation to see which relay primfunc and tir primfunc resulted in using that much of memory. In order to access this, in the PR, you could use relay.build(…)'s output which is lib : GraphExecutorFactory. (Use lib.function_metadata).

2 Likes
#6

Hi @manupa-arm ,

Thank you for this work :slight_smile:

I tried to access this data after compiling the GraphExecutorFactory, but am a bit confused by the output:

The output of function_metadata is a JSON, which seems to represent a dictionary, but the values for each function look like this relay.backend.FunctionInfo(0xcb63c40). How can I access the additional information from this? I am probably missing something…

#7

Hi @max1996 ,

function metadata is of type :

Map<String, FunctionInfo> function_metadata;

When accessed via Python it functions as a dictionary :slight_smile: .

struct FunctionInfoNode : public Object {
  Map<Target, Integer> workspace_sizes;
  Map<Target, Integer> io_sizes;
  Map<Target, Integer> constant_sizes;
  Map<Target, tir::PrimFunc> tir_primfuncs;
  Map<Target, Function> relay_primfuncs;

A FunctionInfo structure should also be visible in Python and again it creates Maps (dicts in Python) that maps for Target → various_info.

Try printing the items, so you should see what I mean. Maybe using a code snippet as follows :

func_metadata = lib.function_metadata    
for func_name, finfo in func_metadata.items():
        print(finfo.workspace_sizes.items())
        print(finfo.tir_primfuncs.items())
1 Like
#8

Ah, yes, thank you. I am not sure, what I did wrong on the first try, but it is working now.

Thank you very much :slight_smile:

1 Like
#9

Hi @manupa-arm ,

I used your system to get info on a wide range of different configurations for conv2d, 2d pool and global pooling layers with regards to workspace size & IO traffic and have 2 questions:

  1. I assume this numbers are in byte? I could not find any information about this
  2. Especially for pooling layers, I sometimes receive negative values for the IO numbers, which cannot be true, right?