TF 1.15 and FusedBatchNormV3

apivovarov · March 3, 2020, 8:08pm

I noticed that TF 1.15 uses FusedBatchNormV3 instead of FusedBatchNorm in models such as Resnet. E.g. Resnet50 uses 49 FusedBatchNormV3

Looks like Relay TF frontend does not support FusedBatchNormV3 yet.

github.com

apache/incubator-tvm/blob/master/python/tvm/relay/frontend/tensorflow.py#L1554


'Equal'                             : _broadcast('equal'),
'Elu'                               : _elu(),
'Erf'                               : AttrCvt('erf'),
'Exp'                               : AttrCvt('exp'),
'ExpandDims'                        : _expand_dims(),
'Fill'                              : _fill(),
'Floor'                             : AttrCvt('floor'),
'FloorDiv'                          : _floordiv(),
'FloorMod'                          : _floormod(),
'FusedBatchNorm'                    : _fused_batch_norm(),
'FusedBatchNormV2'                  : _fused_batch_norm(),
'Gather'                            : _gather(),
'GatherNd'                          : _gather_nd(),
'GatherV2'                          : _gather(),
'Greater'                           : _broadcast('greater'),
'GreaterEqual'                      : _broadcast('greater_equal'),
'Identity'                          : _identity(),
'LeakyRelu'                         : AttrCvt('leaky_relu'),
'LeftShift'                         : AttrCvt('left_shift'),
'Less'                              : _broadcast('less'),
'LessEqual'                         : _broadcast('less_equal'),

It means that if Resnet model frozen .pb file is created using TF 1.15 it can not be compiled with TVM.

jonso · March 4, 2020, 12:35am

@apivovarov, would you be able to contribute this op?

apivovarov · March 4, 2020, 2:48am

V3 has one extra output in comparison to noV or V2 - reserve_space_3 https://www.tensorflow.org/api_docs/cc/class/tensorflow/ops/fused-batch-norm-v3 https://www.tensorflow.org/api_docs/cc/class/tensorflow/ops/fused-batch-norm https://www.tensorflow.org/api_docs/cc/class/tensorflow/ops/fused-batch-norm-v2

We should probably ask people who added fused-batch-norm and fused-batch-norm-v2 to relay TF frontend to look into it.

masahi · March 4, 2020, 4:11am

@jonso Do you think things like this (which seems esoteric to me) is better suited for a custom conversion map, something similar to the one I added for torch? https://github.com/apache/incubator-tvm/pull/4961