We currently have two families of “Runtime” in TVM:
c and c++ runtime – describes the implementation of c_runtime_api.h and c_backend_api.h.
graph, vm, aot runtime – implements Module-Based Model Runtime API (eventually…), which drives model-level execution by invoking a series of operator functions and handling control flow.
Proposal
Let us rename GraphRuntime, VirtualMachine runtime, and AOT Runtime to something else. It’s too confusing to have the word “runtime” everywhere and not everything is a runtime . Here are some options:
The “executor” name sounds good to me. Yeah, I agree that conceptually “runtime” of a language should be a minimal library to support its running, and in TVM, specifically it should be something implementing APIs provided in c_backend_api.h.
right, even the c/c++ runtimes encompass more than you would expect of a true “runtime.” in the AOT proposal, I believe that the c_backend_api.h implementation is all that would be needed.
@zhiics I agree we do not have a specific VMExecutor checked-in. It would become the convention should we add one. For now, we would just change the verbiage in VM docstrings, where needed.
By change APIs, do you mean “change to fit the module-based model runtime?” I don’t know that we’ve settled on a specific interface, but I think there is sentiment that having a single interface for all executors is a good thing.
What is the expectation of the user to choose which “executor” to be compiled for ?
It would be great if it could be unified under relay.build(…), possibly with an additional argument to specify the executor (default can be the “graph” as to not break any downstream deployments).
What is the expectation of the user to choose which “executor” to be compiled for ?
we haven’t settled this. Ideally in my mind, the user could just specify a list of executors, but I’m not sure what the effects of that are on the internals.
Yeah that is a good point. I do think that if we are going with Executor to describe the runtime, we should rename the existing tvm.relay.Executor interface to something e.g. ExecutorWrapper. We could also contemplate the future of relay.create_executor, but should probably do that in a separate interface.