Hi @masahi , I am not quite clear regarding this bias_add and add op folding that you mentioned.
Do you mean after simplification i.e., CanonicalizeOp and SimplifyExpr passes there will finally be an add op? In other words, will the add op (shift from batchnorm) and bias_add be effectively written as 1 “add” op?
It could be an issue for my usecase. As I mentioned in my original question I would need to preserve the conv2d and bias_add ops after batchnorm fold. It is more of a pattern matching requirement rather than an optimization one.
Also, in the cases where bias_add is not present, I would need to have the “shift” from batchnorm to be expressed as an bias_add op.
So, what I intend to achieve is as follows:
case 1:
before: conv2d → bias_add → add (shift from batchnorm) is transformed to:
after transform: conv2d → bias_add ( bias values are changed by add op folding into bias_add op)
case 2:
before: conv2d → add (shift from batchnorm) is transformed to:
after transform: conv2d → biad_add (add op expressed as bias _add op)
Can you please confirm if this seems possible with existing TVM transformation passes?