[RFC] [uTVM] Embedded C Runtime Interface

Development pre-RFC
7

Apologies @areusch, I’ve left this thread alone for a rather long time and wanted to respond to your comment in the memory planning thread to do with generating structs for context. As we discussed on Discord, this inspired me to revisit your previous user-facing API comment:

Looking at this API now, it is definitely feeling like we’d be limiting how user-friendly we can make this API if we use the originally suggested interface. As such, I’ve proposed an implementation in https://github.com/apache/tvm/pull/8280 which uses a standard interface with the four currently known slots (inputs, outputs, memory, context) in a model entrypoint function (currently tvm_default_run as the model is named tvm_default until https://github.com/apache/tvm/pull/8014 lands):

/*!
 * \brief TVM default model input tensors 
 */
struct tvm_default_inputs {
        void* i;
};

/*!
 * \brief TVM default model output tensors 
 */
struct tvm_default_outputs {
        void* output;
};

/*!
 * \brief TVM default model memory blocks 
 */
struct tvm_default_memory {
};

/*!
 * \brief TVM default model device configurations 
 */
struct tvm_default_devices {
};

/*!
 * \brief TVM default model run function 
 * \param inputs Input tensors for the model 
 * \param outputs Output tensors for the model 
 * \param memory Memory blocks for the model to use 
 * \param devices Devices for the model to use 
 */
int tvm_default_run(
        struct tvm_default_inputs* inputs,
        struct tvm_default_outputs* outputs,
        struct tvm_default_memory* memory,
        struct tvm_default_devices* devices
);

The other observation I had from [RFC] Unified Static Memory Planning was that the memory could be specified a bit differently:

  • S1. Specify each type of memory such as parameters and workspaces with differing structures
  • S2. Specify each type of memory such as parameters and workspaces as nested structures
  • S3. Specify the names for these memories only and the user already has the knowledge as to which block they wanted to use

S1 and S2 introduce the difficulty of what do we name each piece and how do we make it easy for users to setup their memory, so for the above example I’ve gone with S3 - it seems to be more straight forward but would like input from @manupa-arm as to whether this will be better or worse :smiley_cat: