[pre-RFC] Compilation Configuration Representation

[Mousius]

Oh wow, I’ve been away for a few days and really appreciate the amount of discussion that’s arrived Thanks @mbs-octoml, @zxybazh, @tqchen, @comaniac, @junrushao and @manupa-arm!

Firstly let’s address a few specifics which may help narrow the discussion slightly:

@junrushao

Last, there are cases where multiple executors working together. For example, if we want to offload some fragments to TensorRT executor, some to CUDA graph, while keep the rest in the VM, then the Relay function could potentially be partitioned into 3 Relay functions that targets to different executors. With composite target, we are able to attach different executors in the Target object in a more explicit way.

There’s an unfortunate overloading of the terms Executor and Runtime which is the inherent risk with a diverse heterogeneous compiler . In this RFC, let’s define the Executor and Runtime as specific to the TVMs Executor and Runtime rather than the implementation of a Target. How a Target gets generated and linked is outside the immediate scope of the TVM Runtime and Executor as they’re designed to invoke the generated Target code.

@mbs-octoml

• I think you want your tvmc cmdline flags to be interpreted as rewrites on CompilerConfig in some uniform way, right?

@junrushao

The RFC text doesn’t mention how it could improve the UX in TVMC command line. However, I would argue that the UX could be improved simply with target tags. For example, on CUDA GPUs, our target tag system supports creating CUDA targets with a single short string.

Thanks @mbs-octoml, I missed some Prior Art here! In tvmc, we have the concept of a configuration as defined in Command Line Configuration Files. CompilationConfig would allow this to be a standard way of defining such as configuration with Targets within it - this meets the needs of the Target tagging which @junrushao and @tqchen are discussing by instead wrapping them into a CompilationConfig that represents the system. Within the Command Line Configuration Files RFC, it defines the <TYPE> and indicates the use of --config for cloud instances. The terminology would shift from a tagged Target to a CompilationConfig here to represent they exist at two different levels of the hierarchy?

@tqchen

• I do not think there is a strong contention on C1, the main point is that the target can be recursive. So a target like the follows is totally OK.

As defined in Migrating Target Attributes to IRModule, splitting the TVM concepts of Target, Runtime and Executor means we can more clearly see what is most relevant to a specific Target. Which means that call-site annotations for Target are limited to only options that are relevant to a specific Target rather than to an IRModule. By virtue of working on this RFCs implementation, although we should still land the implementation agreed in the RFC, it does illustrate how we can better manage the representation of this configuration internally to TVM.

One reason not to motivate this as purely a tvmc concern is that tvmc is the CLI interface to TVM, if a user attempts to use tvmc and then moves to a Python script they should not be re-learning the interface to TVM.

@tqchen

G1: Ability to enable incremental customization (via python API), attach constraints(such as BYOC) and then send back to the build function for further lowering.

This sounds sensible, and also a feature of CompilationConfig is the ability to specify the complete picture of the system which TVM is being built for including all Targets which can be used by all Passes. Specific annotations of storage and execution make sense to be defined at call-sites within the IR rather than at the top level with the IRModule - what CompilationConfig provides is a frame of reference to do those annotations and pick from a variety of Targets and Devices which the IRModule is constrained to. As we continue with Target registered compiler flow customisation, annotating a call-site with a Target will become standardised with the BYOC flow whether partitioned or otherwise to match the expectation you described with partitioned Targets.

@zxybazh

The design of targets list and target host seems really similar to a composite target, can we actually use a single composite target (where each target can be a composite target) with executor and runtime field as compilation config?

This doesn’t rule out the possibility of using a composite Target as a Target in the targets list as we’re not redefining how that works here - rather defining a bounding box for system level configuration within TVM.

@zxybazh

CheckAndUpdateHostConsistency is designed to make sure target.host and target host are consistent along the way. If we have a better machanism to do that since we have target host as a field in compilation configuration, we may further reconsider the usage of target inside of compilation flow.

The end state for this configuration update would be to run a single pass over the CompilationConfig early on to ensure the internal state was correct using CheckAndUpdateHostConsistency which guarantees that subsequent Passes such as device or memory planning are safe making assumptions about the state of the used Targets. Hopefully that clarifies it’s less of a replacement, but more of a consolidation of the logic to early in the compilation flow if these checks are still required We’d still need to have Target annotations within the IR and that Target will therefore have to be stable during compilation.

Where we’re at

Going over this thread a few times, the discussion revolves around:

M0. Split the CompilationConfig from Target

(CompilationConfig)
-> (Target), (Target), (Target)
-> (Executor)
-> (Runtime)

M1. Recursively allowing Target to represent any system

(Tagged Target)
-> (Target), (Target), (Target)
-> (Executor)
-> (Runtime)

It is my opinion, and the motivation behind this RFC, that better defining the CompilationConfig would relieve cognitive load on the user and provide definitions which can be bounded easily. By continuing to use M1 the term Target is increasingly overloaded and difficult for both developers of TVM and more importantly users of TVM. This hierarchical terminology has prior art in large scale cloud frameworks, such as Kubernetes which uses different terminology for Cluster, Deployment, Service, Pod and Container which are all different levels of granularity of computing resources; the decision here is both a UX decision and a practical separation of concerns for both users and developers of Kubernetes.