[External Codegen] Constant tensors in c-codegen

Questions
#21

Thanks @masahi for chiming in. The CCompiler example is only used for demonstration purpose. We intentionally made it simple to handle cases like constant. For real external codegen tools/compilers, you may have your own ways to handle the constant pool, and it is very backend compiler dependent. For example, you can serialize it in the codegen and deserialize it when you load the library as @masahi mentioned. TVM compilation tools for graph runtime and the VM are all doing this. The TensorRT work leverages the serialization facility of ObjectRef provided by TVM. Also, you can probably have something compatible to dlpack which would essentially be able to point to the constant data for in memory execution.

1 Like
#22

Hi @masahi, that is correct. I am using TVM’s native json serialization API to serialize the relay subgraphs during codegen and deserialize it in the runtime for my TRT integration.

I am curiously what binary format you are using to serialize relay subgraphs? I have found that TVM’s JSON serialization is not as stable as I would have liked since any change in the TVM codebase to VisitAttrs will change the format.

#23

@trevor-m Thanks for confirming. I can’t talk about specific, but let’s just say any cpp serialization lib should be able to serialize/deserialize structs into a binary blob, and I am just using one of them.

Note that I’m not serializing Relay subgraph as it is, but some structs that get converted from Relay and amenable for serialization by a third party lib.

#24

@masahi I see, thanks for sharing. I also thought about an approach like this. It seems like a lot of additional overhead to create and maintain a whole new IR for serialization. Since it seems to be the case that many external codegens (not just TRT) will need to do something like this, I wonder if we could provide some infrastructure in TVM for this problem.

#25

In my case, these intermediate structs are strongly tied to our executor. They are plain structs, so much easier to work with than full blown relay IR. So for me they are not really overhead.