Questions about TVM executors and its APIs

Questions
3

Hi @yuchenj, thank you for your kind reply! I want to confirm some follow-up questions and report some issues.

It seems in your mentioned line (Optimize function, build_module.cc:303), a set of (default I think) passes are given. Is that the case: in BuildRelay we will collect some default passes (those in Optimize) into a Sequence, and apply them if pass-level opt_level >= context-level opt_level (filtered in SequentialNode::operator()).

I asked about these questions b.c. I think some tutorial did not show good practice about APIs.

When using relay.build_module.create_executor API, it seems the compilation (BuildRelay) is done in evaluate(). (evaluate() calls interpreter.Executor._make_executor, GraphExecutor._make_executor calls build, and Interpreter._make_executor call some extra optimization passes.)

This means we should call evaluate() inside PassContext otherwise some default passes will be filtered.

import tvm
import tvm.relay as relay
from tvm.relay import testing

def example():
    data = relay.var("data", relay.TensorType((1, 3, 512, 512), "float32"))
    weight = relay.var("weight")
    bn_gamma = relay.var("bn_gamma")
    bn_beta = relay.var("bn_beta")
    bn_mmean = relay.var("bn_mean")
    bn_mvar = relay.var("bn_var")

    simple_net = relay.nn.conv2d(
        data=data, weight=weight, kernel_size=(5, 5), channels=32, padding=(1, 1)
    )
    simple_net = relay.nn.batch_norm(simple_net, bn_gamma, bn_beta, bn_mmean, bn_mvar)[0]
    simple_net = relay.nn.relu(simple_net)
    simple_net = relay.Function(relay.analysis.free_vars(simple_net), simple_net)

    return testing.create_workload(simple_net)

if __name__ == '__main__':
    mod, params = example()
    target = tvm.target.Target('llvm')
    dev = tvm.cpu()
    with tvm.transform.PassContext(opt_level=4): 
        executor = relay.build_module.create_executor("graph", mod, dev, target)

    # Here `evaluate()` is called outside `PassContext` like the following tutorial did:
    # https://tvm.apache.org/docs/tutorials/frontend/from_onnx.html#compile-the-model-with-relay
    tvm_out = executor.evaluate()(
        tvm.nd.empty(shape=(1, 3, 512, 512), 
        device=dev, 
        dtype='float32'), **params)

Doing so, the opt_level=4 passes will not be applied. But if we do:

    with tvm.transform.PassContext(opt_level=4): 
-        executor = relay.build_module.create_executor("graph", mod, dev, target)
+        executor = relay.build_module.create_executor("graph", mod, dev, target).evaluate()

-    tvm_out = executor.evaluate()(
+    tvm_out = executor(

We finally see some logging like “… tvm/src/relay/ir/transform.cc:133: Executing function pass : CombineParallelConv2d with opt level: 4”.

Such tutorials are:

I hope my report might help enhance the tutorial.