Could we know detail about the applied optimization level?

Questions
#1

I am learning about tvm optimization.
I am interested in opt_level with tvm.transform.PassContext(opt_level=3).

I have three questions.

  1. Could you tell me what optimization technique is used when opt_level is specified as 3?
  2. If possible, could we use print to show optimization methods?
  3. Could you give me some advice when learning the optimization methods?

I have visualized the AlexNet computation graph to learn the optimization techniques.
I plan to understand graph optimization by focusing on visualizing computational graphs.

  1. I want to know the names of the methods used when the optimization level is specified.
    I feel that multiple methods are used at each optimization level.
  2. I will visualize the neural network graph using the tvm.transform.Sequential function specifies any optimization method.
  3. I will find what optimization has been performed on the visualized neural network.

The following provides the environment in which I am experimenting.

#2

To partially answer your first question, you can see some discussion of this in my thread here. This is not a complete set of passes, but a given pass is applied if opt_level >= [number].

OPT_PASS_LEVEL = {
    "SimplifyInference": 0,
    "OpFusion": 1,
    "FoldConstant": 2,
    "FoldScaleAxis": 3,
    "AlterOpLayout": 3,
    "CanonicalizeOps": 3,
    "CanonicalizeCast": 3,
    "EliminateCommonSubexpr": 3,
    "CombineParallelConv2D": 4,
    "CombineParallelDense": 4,
    "CombineParallelBatchMatmul": 4,
    "FastMath": 4
}

E.g., OpFusion is applied when opt_level >= 1.

2 Likes
#3

I asked a similar question a couple of days ago. The first answer and my futher findings might be helpful.

https://discuss.tvm.apache.org/t/default-relay-passes-in-pathcontext/12898

I’m not sure if the order listed by @Wheest is still correct since the documentation states that build_config is deprecated and replaced by the opt_level stored in the PassInfo. Howerver the explanation still give the idea on how the passes in a sequential are checked against the opt_level of the PassContext. So from my understanding you have to check the passes and their opt_level individually and then check wheather they are applied in the two functions shown in my post.

1 Like