Here’s the problem. I want to fuse the QKV attention operation in a transformer network, I mean I want to make a big operator which contains the whole operations in the self-attention struct.
How can I do this? Existing Relay Passes do not support this sort of fusion. Should I write a new Pass or there’s another way to solve this problem?
Or does Relay support modify computation graph without using Pass?