[RFC]Add a PaddlePaddle Frontend

Development pre-RFC
#1

Adding PaddlePaddle Frontend for TVM

Hi, all, I am one of PaddlePaddle’s developer, leading the projects as follow list,

I have noticed there are lots of PaddlePaddle’s user mentioned the demand for TVM. After almost 2 months studying and experiment, I found that TVM does show its high performance after auto tune the kernel. So here I propose this RFC to add a frontend of PaddlePaddle, which will give more option for PaddlePaddle’s users, also extend the TVM’s compatibility and expand user group.

Background & Motivation

PaddlePaddle, an independent R&D deep learning platform in China, has been officially open-sourced to professional communities since 2016. It has been widely adopted by a wide range of sectors including manufacturing, agriculture, enterprise service, and so on while serving more than 2.3 million developers. With such advantages, PaddlePaddle has helped an increasing number of partners commercialize AI.

Currently, PaddlePaddle has built a prosperous technological ecology, there are more than 500 models developed by official organization or outside developers, including CV/NLP/OCR/Speech, for more details we can refer to the following links,

After upgrading to 2.0, PaddlePaddle supported imperative programming similar with PyTorch, but a mechanism of Dynamic to Static is provided, which can export PaddlePaddle model as graph representation and more friendly for deployment, the following example code shows how to export a PaddlePaddle model.

import paddle
import paddlehub
model = hub.Module(name="resnet50_vd_imagenet_ssld")
input_spec = paddle.static.InputSpec(
        [None, 3, 224, 224], "float32", "image")
paddle.jit.save(model, "model/infer", input_spec=[input_spec])

Currently, PaddlePaddle’s deployment is supported by Paddle Inference/Paddle Lite/OpenVINO/Tengine/Adlik now. We have noticed there are lots of developers converting PaddlePaddle model to ONNX format for TVM’s supporting, but only part of models can be converted due to the lack of ONNX operators.
Based on this background, we proposed this RFC to add PaddlePaddle frontend for TVM, improve usability and extend more models support for PaddlePaddle’s users.

Implementation Approach

The whole process of PaddlePaddle frontend importing model is divided into:

  1. Read Model: The frontend supports PaddlePaddle’s inference model format which is exported as graph based model by PaddlePaddle’s Dynamic to Static mechanism. The model contains 2 files that store the model structure and parameters separately, We use paddle.static.load_inference_model to load the model files;
  2. Operator Conversion: After we load inference model, the remaining work is traversing Program structure, and convert operators one by one.

Finally, we can import a PaddlePaddle model as follow example codes:

from tvm import relay
import paddle
model = paddle.jit.load('model/infer')

input_shape = {'image': [1, 3, 224, 224]}
mod, params = relay.frontend.from_paddle(model, shape_dict=input_shape)

Work Done

  1. List of supported operators
  • arg_max
  • assign
  • batch_norm
  • cast
  • concat
  • conv2d
  • cumsum
  • depthwise_conv2d
  • dropout
  • elementwise_add
  • elementwise_div
  • elementwise_mul
  • elementwise_sub
  • equal
  • exp
  • fill_any_like
  • fill_constant
  • gelu
  • hard_sigmoid
  • hard_swish
  • layer_norm
  • leaky_relu
  • lookup_table_v2
  • matmul
  • matmul_v2
  • pool2d
  • relu
  • reshape2
  • scale
  • shape
  • slice
  • softmax
  • tanh
  • transpose2
  • unsqueeze2
  1. List of supported models Currently, we only run testing on the following models, in this quarter, we will expand the model list to 50+
  • ResNet50
  • MobileNetV2
  • MobileNetV3
  • Bert-Base-Uncased

TODO

  • Network forward and single operator testing(tvm/tests/python/frontend/paddlepaddle)
  • Support more operators, such as deformable_conv/multiclass_nms/lstm/gru
  • Support control flow operators, mainly about while_loop/if
  • Support quantized model, include quantized models from PaddleDetection/PaddleClas/PaddleSeg

The testing code will commit in a week, and for the remaining todo list, we have make an established plan as below,

  • About 200 operators will be supported in this quarter
  • Control flow operators will be supported in this year
  • Quantized model will be supported in this year

Here is the pull request Add a PaddlePaddle Frontend by jiangjiajun · Pull Request #8645 · apache/tvm · GitHub , If there’s any question, I will reply here in time. Also feel free to concat my email(jiangjiajun@baidu.com) directly, thanks~

@tqchen @FrozenGene

7 Likes
#2

Thank you @jiangjiajun . We started to adopt a new RFC process GitHub - apache/tvm-rfcs: A home for the final text of all TVM RFCs., after some preliminary discussions here, we can move to send a formal PR to the repo

#3

Thanks @tqchen

I just found the new RFC process, the PR of RFC will be send in tomorrow.

1 Like
#4

Here is the pull request of RFC [RFC][Frontend] Add a PaddlePaddle Frontend by jiangjiajun · Pull Request #19 · apache/tvm-rfcs · GitHub