Thanks for the explanation. In terms of expected behavior, the result is different when I use other passes like FoldConstant (which is an opt_level=2 pass). Regardless of which method is used the FoldConstant pass applies the same transformation. Should FoldScaleAxis be altered so it behaves in the same way?
Here is an example for FoldConstant:
import tvm
from tvm import relay
import numpy as np
def create_model():
data = tvm.nd.array(np.array([1.0, 2.0, 3.0]))
const = relay.const(data)
concat = relay.op.concatenate([const, const], axis=0)
mod = tvm.IRModule.from_expr(concat)
return mod
mod = create_model()
mod_0 = relay.transform.FoldConstant()(mod)
with tvm.transform.PassContext(opt_level=2):
mod_1 = relay.transform.FoldConstant()(mod)
seq = tvm.ir.transform.Sequential([relay.transform.FoldConstant()])
with tvm.transform.PassContext(opt_level=2):
mod_2 = seq(mod)
assert tvm.ir.structural_equal(mod_0, mod_1)
assert tvm.ir.structural_equal(mod_1, mod_2)
I don’t think so. FoldConstant behaves the same for both methods just because it is opt_level=2, which is the default level. Passes such as FoldScaleAxis are assigned to higher level for reasons, so you should always use the PassContext to specify the desire opt_level when invoking passes.
In the case of CanonicalizeOps which has an opt_level=3 we get the same behavior as FoldConstant where we don’t have to specify the opt level for it to work in the case of mod_0 = relay.transform.CanonicalizeOps()(mod)
Example Code:
import tvm
from tvm import relay
import numpy as np
def create_model():
x = relay.var("x", shape=(1, 64, 56, 56))
bias = relay.var("bias", shape=(64,))
weight = relay.var("weight", shape=(64, 64, 3, 3))
conv = relay.nn.conv2d(x, weight, channels=64, kernel_size=(3, 3), padding=(1, 1))
bias_add = relay.nn.bias_add(conv, bias)
mod = tvm.IRModule.from_expr(bias_add)
return mod
mod = create_model()
mod = relay.transform.InferType()(mod)
mod_0 = relay.transform.CanonicalizeOps()(mod)
with tvm.transform.PassContext(opt_level=3):
mod_1 = relay.transform.CanonicalizeOps()(mod)
seq = tvm.ir.transform.Sequential([relay.transform.CanonicalizeOps()])
with tvm.transform.PassContext(opt_level=3):
mod_2 = seq(mod)
assert tvm.ir.structural_equal(mod_0, mod_1)
assert tvm.ir.structural_equal(mod_1, mod_2)
Sorry my previous description wasn’t accurate. If you directly invoke a pass such as relay.transform.FoldConstant()(mod), then it may or may not consider the current opt_level. It depends on whether the pass is a function pass or a sequence pass.
For example, FoldConstant and CanonicalizeOps are function passes. In this case, if you invoke them directly, it will be applied for sure.
On the other hand, FoldScaleAxis is a sequence pass:
It means the following two methods are equivalent, and both methods are affected by the current opt_level.
Do you think it would be a good idea to modify the description of the FoldScaleAxis pass? I would add that that FoldScaleAxis is a sequential pass and must be placed within a PassContext of 3 for all passes contained within it to execute.