Data layout awareness in BYOC

Troubleshooting
#1

An issue occurs in BYOC that x86 scheduling Strategy implemented NCHW -> NCHWc conversion in data layout before and after entry into conv2d. The graph looks sth like this:

… -> layout_transform -> conv2d -> bias_add -> layout_transform -> …

BYOC flow will write out function calls to customer implemented kernels (conv2d, bias_add) here, however, BYOC flow isn’t aware of the layout conversion at all. Since customers take full responsibility in BYOC subgraph, the data layout transform incurred in relay optimization caused wrong data fed into operator functions.

Is this a flow bug for BYOC to be resolved?

#2

IIUC, layout transform is not a part of Relay optimization but TE optimization, meaning that the alter layout happens after lowering Relay graph to TE graph, which won’t happen to the offloaded subgraphs. For example,

(conv2d -> add) -> conv2d -> add

In this Relay graph, we only offload the first conv2d plus add. Then the graph shuld be like the following after the compilation

(external kernel) -> layout_transform -> conv2d (NCHWc) -> add -> layout_transform

Could you provide a more concrete example for your use case to help locate the problem?

#3

@comaniac, thanks for your prompt reply. Revisiting my case, I think you are right.

I’m hacking in TVM to have disabled fuseOps pass in relay to allow a unit-level test of BYOC bias_add operator can be singled out. See below graph snippet, function (74) is the bias_add that be going down the BYOC flow. In deed, layout tranformations (func(84) and func(77)) are guarding conv2d as you’ve described.

func(81) conv2d_NCHWc

func(84) and func(77) layout transform

func(74) bias_add

lenet_add

My unit test failed unfortunately and I’m still debugging it (that’s why I raised the potential layout transform problem).

Question now becomes: how can we provide params in right layout in such case? Constant(idx=87) are the network coefficients, Constant(idx=90) are the bias_add coeffs. Thanks.

#4

If there are constants, then you don’t (and cannot) provide values as params. The constant values will be maintained by metadata module and you shouldn’t have to worry about their shapes.