[RFC] [uTVM] Embedded C Runtime Interface

Development RFC
5

@Mousius thanks for your reply! I think we are nearly aligned here.

I think we’ll likely want a “model header” for the extern TVMModel my_model eventually anyway, so I’d be interested to hear of use cases that aren’t viable with this as defines rather than in the struct; such use cases aren’t immediately coming to mind.

I think #define should definitely suffice for anything used in the downstream program. The main other use cases are those which consume this stuff outside of the C compiler (e.g. Python tools which analyze the generated code). Those I think would prefer JSON. I think so long as each parameter in the header is represented in the metadata.json, this concern is addressed. It would be nice to consider a standard way to map metadata.json keys to #define name.

Default Model Compilation

Some points of feedback:

  • I think we should also consider a use case with multiple model inputs and how a user should identify the order to populate inputs.

  • Similar suggestion for outputs.

  • I do think here, inputs and outputs could be a struct which would allow for:

    inputs.my_input = model_input

    rather than

    inputs[TVM_my_model_my_input_INDEX] = model_input

  • Presumably TVM_my_model_INPUT_SIZE would similarly need to include the name of the input.

  • TVMExecute’s return code should be checked. It would be great to consider the error case here too (generally, I imagine it’s just printing the error code and/or TVMGetLastError() (see discussion here) and cc @stoa.

Custom Workspace Compilation

I think this seems fine, don’t have anything to add over the other thing here (except see comment on TVMContext below).

Other commentary

How this is represented in the JSON wasn’t something I was intending to address here but I agree that this should be accessible in the Model Library Format for other toolchains to interact with.

Okay, we can iterate on this in future RFCs/PRs.

Could you expand on what you mean here? In a default flow I’d expect the workspace to be generated and codegen to pass that to the underlying operators, that means it would exist by default in flash for “Default Model Compilation”.

Here I just mean the TVMContext struct itself. I’m not sure its necessary to place it on the stack when everything is a global.

My core motivation here is to provide a stable API for interacting from a user application, so you can run:

TVMExecute(<model>, <inputs>, <outputs>, <context>);

Which should work as we move forwards with TVM and users recompile their models however the model is internally represented. Similarly, however we wish to structure the context is hidden from the user and we can augment it with the use-cases as they arrive, but still be able to run:

TVMSetWorkspaces(<context>, <list of workspaces>);

In such an environment, each piece of orthogonal information can exist on the TVMContext struct as we grow it. Do you have an alternative you can outline?

I think that’s a reasonable goal. I’m okay keeping TVMContext so long as we add internal organization, then:

struct TVMContext {
  struct TVMMemoryLayout {
    uint8_t* workspaces;
  } memory_layout;
  struct TVMDeviceMapping {
    tvm_model_name_device_id_t device_id;
    tvm_device_t device_instance;
  } *devices;
};