[BYOC, CUTLASS] Dealing with Constants in C source-gen based BYOC

[masahi]

@comaniac

This mechanism is used for the case that a BYOC backend attempts to manage the constant values with certain processes, such as layout transform

CUTLASS does seem to support specialized layouts for gemm / conv2d. If we want to make use of them and if the layout transform cannot be expressed by relay.transform, then I think we really need to take in Constants.

@comaniac

IMHO, we could have a specialized mechanism for C codegen to manage constants. For example, we could let C codegen serialize the constants to a separate artifact file, and encapsulate it along with the generated/compiled engines, and load them to the memory at the first execution

This makes sense, the problem I imagine would be that C-codegen doesn’t really have a proper runtime. So I guess all logic for managing constants will be written in a big string and compiled together with the actual offload calls. I don’t want to write code like that

So I think we need to think about switching to Json runtime if a need for a proper handling of Constants ever comes up. The difficulty I see is that unlike C-codegen based BYOC or TensorRT, we have to manage compilation and loading of the compiled lib ourselves (call nvcc directly from the Json runtime and use dlopen etc to retrieve a handle to a compiled function).

Maybe what we need is something like NVRTC for cutlass.