Op fusion in tvm with nn.bitpack op

Questions
#1

Hi Everyone.

When I am using bitpack op in a model, sometimes it gets fused with cast op. above, like (other_ops+cast+bitpack) and sometimes it doesn’t added to the fused group such that (other_ops+cast), (bitpack).

I did not get how the tvm operator fusion works in this scenario? any help.

#2

It’s hard to explain why. The fuse rule is too complex.

reg.register_pattern("nn.bitpack", OpPattern.INJECTIVE)

maybe your could read code src/relay/transforms/fuse_ops.cc

void RunFuse(const IndexedForwardGraph& graph, const DominatorTree& post_dom_tree, int phase);
#3

Thank you for the Information.

My graph consists of these nodes in sequential. [multiply, add, relu, cast, bitpack, custom-op, cast, multiply, add, relu, cast, bitpack, custom-op, cast, add]

After fusion, it is transformed to four ops as below

  1. multiply+add+relu+cast+bitpack
  2. custom-op+cast+multiply+add+relu+cast
  3. bitpack
  4. custom-op+cast+add

In node1, bitpack is fused to the cast op before it but in 3, it is not added to the fused op.

reg.register_pattern(“custom-op”, OpPattern.OUT_ELEMWISE_FUSABLE)

#4

OUT_ELEMWISE_FUSABLE seems could only fuse with ELEMWISE op. cast is kElemWise. add kBroadcast, but if add inputs shape and outputs shape is same, add will become ElemWise. so you can see bitpack not fused with before 2. relu kElemWise.

image
image816×140 10.4 KB

#5

Thank you.

I want to fuse the cast op followed by bitpack op. Is there any way to always fuse the bitpack op to the elementwise ops above it like cast etc. ?

#6

insert stop_fusion op before cast.

#7

Thank you.

Original Graph:

  1. cast
  2. bitpack
  3. custom-op+cast+add+multiply+add+relu+cast
  4. bitpack
  5. custom-op+cast+multiply+add+relu+cast
  6. bitpack

after adding the stop_fusion node before cast op, it is transformed to:

  1. cast+bitpack
  2. custom-op
  3. cast+add+multiply+add+relu
  4. cast+bitpack
  5. custom-op
  6. cast+multiply+add+relu
  7. cast+bitpack

The issue before inserting stop_fusion is that the bitpack op doesn’t get fused with the cast. But after adding, the custom-op became the single op without getting fused.

I am looking to transform the original graph to:

  1. add+relu+cast+bitpack
  2. custom-op+cast+add+multiply+add+relu+cast+bitpack
  3. custom-op+cast+multiply+add+relu+cast+bitpack

But issue is, two op’s with above kBroadcast level can’t be fused:

    if (lhs > kBroadcast && rhs > kBroadcast) {
        LOG(FATAL) << "Cannot merge two complex group together";
    }

Any solution to achieve the above transformation? thanks

#8

Are u always insert stop_fusion op before cast?

#10

Thank you @chenugray

I want to fuse the two ops of kind Out_Elementwise_Fusable and Injective. But existing fuse_ops.cc does not allow.

If we see bitserial convolution, bitpack is injective and convolution is out_elementwise_fusable and both ops work as a single op in bitserial_conv2d but If we fuse the two ops (i.e nn.bitpack and custom_op_conv2d) explicitly, it throws an error as “Cannot merge two complex group together”?

I didn’t get how the schedule differs from the individual ops to the ops if they fused.

Thank you