[SOLVED] MergeComposite and diamond-shaped structure

masahi · April 15, 2020, 9:18am

I’m having hard time matching a pattern like below. It seems one path, lhs of qmul to qnn.conv2d, is correctly matched, but it fails when it continues to the rhs of qmul. Is this pattern supposed to be work?

Specifically, it fails at this line during matching the rhs clip.

github.com

apache/incubator-tvm/blob/master/src/relay/transforms/merge_composite.cc#L58


    // it with the value at 'root'
    auto free_var = Var(pattern->name_hint(), Type());
    var_map->Set(pattern->name_hint(), Array<Expr>({free_var, root}));
    return std::move(free_var);
  } else {
    // if we have encountered this var already, return the free var that was created
    auto vars = (*var_map)[pattern->name_hint()];
    auto free_var = vars[0];
    auto graph_expr = vars[1];
    // make sure to first check they both map to the same node in the graph
    if (graph_expr != root) {
      return Expr();
    }
    return (*var_map)[pattern->name_hint()][0];
  }
}


Expr ExtractPattern(const Constant& pattern, const Expr& root,
                    Map<std::string, Array<Expr>>* var_map) {
  return root;
}

   qnn.conv2d
      *
      *
   bias_add
      *
      *
   requantize    
      *
      *
     clip    
      *
      *
     cast
    *     *
    * 	    *
    *	    clip
    *	    *
    *     *
     qmul

mbaret · April 15, 2020, 9:23am

I’d expect it to work (or at least, it’s a bug if it doesn’t ). Could you please show me what code you’re using to define the pattern?

masahi · April 15, 2020, 9:32am

The pattern is from make_qnn_conv_hswish_pattern below. It is an easier case of quantized hswish following qconv.

def make_qnn_mul_pattern(lhs=None, rhs=None):
    if lhs is None:
        lhs = relay.var('lhs')
        rhs = relay.var('rhs')
    lhs_scale = relay.var('lhs_scale')
    lhs_zero_point = relay.var('lhs_zero_point')
    rhs_scale = relay.var('rhs_scale')
    rhs_zero_point = relay.var('rhs_zero_point')
    output_scale = relay.var('output_scale')
    output_zero_point = relay.var('output_zero_point')
    return relay.qnn.op.mul(lhs, rhs, lhs_scale, lhs_zero_point,
                            rhs_scale, rhs_zero_point, output_scale,
                            output_zero_point)

def make_qnn_conv_pattern(with_pad=False):
    x = relay.var('x')
    y = relay.var('y')
    x_zp = relay.var('x_zp')
    y_zp = relay.var('y_zp')                                             
    x_scale = relay.var('x_scale')                                                                                                                
    y_scale = relay.var('y_scale')
    kernel_size = (0, 0)                                                 
    channels = 0
                                                                                                                                                  
    if with_pad:                                                                                                                                  
        inp = relay.op.nn.pad(x, pad_width=((0, 0), (0, 0), (1, 1), (1, 1)))
    else:                                                                                                                                         
        inp = x

    y_scale = relay.var('y_scale')
    kernel_size = (0, 0)
    channels = 0

    if with_pad:
        inp = relay.op.nn.pad(x, pad_width=((0, 0), (0, 0), (1, 1), (1, 1)))
    else:
        inp = x

    qnn_conv = relay.qnn.op.conv2d(inp, y, x_zp, y_zp,
                                   x_scale, y_scale, kernel_size, channels)
    bias_var = relay.var("bias")
    conv_out = relay.op.nn.bias_add(qnn_conv, bias_var)
    requantized = relay.qnn.op.requantize(conv_out,
                                          relay.var("requant_scale"),
                                          relay.var("requant_zp"),
                                          relay.var("output_scale"),
                                          relay.var("output_zp"),
                                          out_dtype="int32", axis=1)
    clip = relay.op.clip(requantized, 0, 255)
    cast = relay.op.cast(clip, dtype="uint8")
    return cast


def make_qnn_conv_hswish_pattern(with_pad=False, with_add=True):
    qconv_pat = make_qnn_conv_pattern(with_pad=with_pad)
    clip = make_clip_pattern(qconv_pat)

    if with_add:
        assert False
        pass # TODO
    else:
        return make_qnn_mul_pattern(qconv_pat, clip)

masahi · April 15, 2020, 9:47am

Note that if I cut the above pattern at the end of cast to get new patterns, one for qconv + requantize and another for clip + qmul, each of them can match correctly. It breaks if I compose them into a single pattern

UPDATE: It seems when the condition graph_expr != root fails, they are pointing to different constants. One of them is qmul’s input scale, and the other is qmul’s output scale.

masahi · April 15, 2020, 10:10am

SOLVED!! I was using the same var name for requantize and qmul’s output scale parameter.

Var name is used for identification during merge composite, so when it visits qmul’s output_scale var, it finds that it has already visited this var name but the value of constants are wrong, so it complains.