# \[RFC\] Differentiable tensor expression (Create and verify backward op automatically)

**URL:** <https://discuss.tvm.apache.org/t/rfc-differentiable-tensor-expression-create-and-verify-backward-op-automatically/7960>\
**Category:** RFC\
**Created:** [September 21, 2020, 2:15pm UTC](https://discuss.tvm.apache.org/t/rfc-differentiable-tensor-expression-create-and-verify-backward-op-automatically/7960 "2020-09-21T14:15:28Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![wrongtest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wrongtest/32/2008_2.png) [@wrongtest](https://discuss.tvm.apache.org/u/wrongtest)\
**Post date:** [September 21, 2020, 2:15pm UTC](https://discuss.tvm.apache.org/t/rfc-differentiable-tensor-expression-create-and-verify-backward-op-automatically/7960/1 "2020-09-21T14:15:28Z")

</div>

As there are more and more demands on TVM’s training support, one of the most tedious but important work is to write backward implementation for operators. It may take great benefit if we can provide automation tools to help this process. Such tool can serve in two functionalities:

- Automatically create backward definition from forward definition
- Check gradient given forward and backward definition

Traditional deep learning framework (perhaps Theano except 😉 ) conduct auto back-propagation on op graph level, that is, they have to implement one backward op given one forward op. Theoretically there should be 10000 backward op definitions if they have 10000 forward ops.

For TVM however, there is an opportunity that we may conduct back-propagation on tensor expression level. Tensor expression operations are much less than whole neural network operators set, thus it will greatly reduce human work on higher level (relay op).

### Backward tensor expression generator

#### Interface

Since tensor expression defines how to compute output from input symbolically, we can just try apply back-propagation rule to it. eg, we can provide utility interface like

```python
def auto_backprop(inputs: List[Tensor], output: Tensor) -> (List[Tensor], Tensor):
    """
    Given input tensor list and output tensor, generate backward computation.
    - The inputs are the placeholder representing the gradient respect to original output and some other necessary original tensors.
    - The outputs are gradients respect to each of the original inputs.
    """
    pass

```

Now if we have already defined some forward computation, then we can extract a “default” backward computation definition:

```python
x = te.placeholder((n, k))
y = te.placeholder((m, k))
z = te.compute((n, m), ...)
  
((grad_x, grad_y), grad_z_placeholder) = te.auto_backprop((x, y), z)
sched = te.create_schedule(grad_x.op)
# Do schedule and tune backward ops...

```

The transformation should happens before create\_schedule(), since generally forward & backward definitions are different and may not share same optimization strategies.

We can wrap this sort of utility in topi and relay, where we can try best to provide default backward op definitions automatically without hand-written definition. Some pros and cons are listed below:

- Pros
  - Avoid hand-written work for at least some portion of operations.
  - Auto generated definition maybe more robust on boundary behaviors and corner cases.

- Cons
  - It is not all-powerfull. Not all operators can be automatically backward.
  - Some optimization hint may lose (backward of matmul is also matmul, backward of conv2d is also conv2d)

#### Transformation logic

At the beginning we may just focus on `te.compute()`, and do not support for tensor intrinsic / hybrid / extern.

- `te.compute()`

- `te.scan()` is also an interesting operation valuable to support back-propagation, with which we can get backward implementations of RNN/LSTM/GRU directly.

### Gradient checking between forward && backward ops

Given forward and backward implementation pair, we can verify the correctness with approximate gradients. This help developer to detect implementation error on general and corner cases. One of the methods is well described in [https://datascience-enthusiast.com/DL/Improving\_DeepNeural\_Networks\_Gradient\_Checking.html](https://datascience-enthusiast.com/DL/Improving_DeepNeural_Networks_Gradient_Checking.html)

---

<div class="post-metadata">

**Author:** ![junrushao](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/junrushao/32/205_2.png) [@junrushao](https://discuss.tvm.apache.org/u/junrushao)\
**Post date:** [September 21, 2020, 6:22pm UTC](https://discuss.tvm.apache.org/t/rfc-differentiable-tensor-expression-create-and-verify-backward-op-automatically/7960/2 "2020-09-21T18:22:31Z")

</div>

Hey @wrongtest,

Thank you for the RFC! Just wondering how it compares with the previous AD RFC ([[RFC] Bring in Tensor Expression Autodiff](https://discuss.tvm.apache.org/t/rfc-bring-in-tensor-expression-autodiff/5987)) ?

Thanks!

---

<div class="post-metadata">

**Author:** ![wrongtest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wrongtest/32/2008_2.png) [@wrongtest](https://discuss.tvm.apache.org/u/wrongtest)\
**Post date:** [September 22, 2020, 2:08am UTC](https://discuss.tvm.apache.org/t/rfc-differentiable-tensor-expression-create-and-verify-backward-op-automatically/7960/3 "2020-09-22T02:08:25Z")

</div>

Glad to see autodiff is already in progress! I think this rfc can be withdrew since this is exactly what autodiff is doing.

Now I am very curious about current progress of autodiff with some questions.

- If I have some common neural network structure such as resnet50 at hand, can I just use autodiff to get backward computation graph?
- Is there some description about common ops which can be coveraged by autodiff?
- Can te.scan() be supported?

---

<div class="post-metadata">

**Author:** ![junrushao](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/junrushao/32/205_2.png) [@junrushao](https://discuss.tvm.apache.org/u/junrushao)\
**Post date:** [September 22, 2020, 4:48am UTC](https://discuss.tvm.apache.org/t/rfc-differentiable-tensor-expression-create-and-verify-backward-op-automatically/7960/4 "2020-09-22T04:48:13Z")

</div>

CC: @yzhliu the major contributor of this feature

---

<div class="post-metadata">

**Author:** ![yzhliu](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/yzhliu/32/28_2.png) [@yzhliu](https://discuss.tvm.apache.org/u/yzhliu)\
**Post date:** [September 23, 2020, 4:49am UTC](https://discuss.tvm.apache.org/t/rfc-differentiable-tensor-expression-create-and-verify-backward-op-automatically/7960/5 "2020-09-23T04:49:25Z")

</div>

> [@wrongtest](#):
>
> If I have some common neural network structure such as resnet50 at hand, can I just use autodiff to get backward computation graph?

graph-wise I think you can refer to [relay.transform.gradient](https://github.com/apache/incubator-tvm/blob/master/python/tvm/relay/transform/transform.py#L713) and as you lower the differentiated graph, you may leverage the tensor-level autodiff ([te.gradient](https://github.com/apache/incubator-tvm/blob/master/python/tvm/te/autodiff.py#L22)). Though tensor gradients now are mostly manually written.

> [@wrongtest](#):
>
> Is there some description about common ops which can be coveraged by autodiff?

You may refer to [test cases](https://github.com/apache/incubator-tvm/blob/master/tests/python/unittest/test_te_autodiff.py)

> [@wrongtest](#):
>
> Can te.scan() be supported?

currently it is not supported.