# Implement Convolution Using TVM

**URL:** <https://discuss.tvm.apache.org/t/implement-convolution-using-tvm/4393>\
**Category:** Application\
**Created:** [October 20, 2019, 4:05am UTC](https://discuss.tvm.apache.org/t/implement-convolution-using-tvm/4393 "2019-10-20T04:05:57Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ziangw2](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/z/3d9bf3/32.png) [@ziangw2](https://discuss.tvm.apache.org/u/ziangw2)\
**Post date:** [October 20, 2019, 4:05am UTC](https://discuss.tvm.apache.org/t/implement-convolution-using-tvm/4393/1 "2019-10-20T04:05:57Z")

</div>

**Hi there,**

**I am a first-time TVM user. I am also not very familiar with CUDA GPU programming. I want to write a tensor convolution routine using tvm. I wish the routine to be at least somewhat optimized. Here is a specification of my problem.**

**1. All the tensors are in NCHW form.**** 2. Vertical and horizontal padding and stride are required. ****3. The code will be executed on an NVIDIA GPU with CUDA, cudnn, cublas etc. support.**

**I am testing a new way of doing convolution, so I really need a base implementation to run my experiment. Built-in convolution routines such as cuDNN are too optimized and are not good baselines for me. I tried to use cutlass but it is too complicated.**

**By reading online tutorials like [How to optimize convolution on GPU](https://docs.tvm.ai/tutorials/optimize/opt_conv_cuda.html) and [Tuning High Performance Convolution on NVIDIA GPUs](https://tkat0.github.io/html/tvm/android-tutorial/tutorials/autotvm/tune_conv2d_cuda.html), I assemble the following code.**

> import numpy as np import tvm

> “”" The computatin we are trying to support is: void\* tensorConvolution(void\* input, void\* filter, &nbsp;&nbsp;&nbsp;&nbsp;int vertical\_pad, int horizontal\_pad, &nbsp;&nbsp;&nbsp;&nbsp;int vertical\_stride, int horizontal\_stride, &nbsp;&nbsp;&nbsp;&nbsp;int conv\_mode, int conv\_groups); “”"

> ######################## # Step 1. Describe the computation ########################

> # Tensor input and filter dimensions input\_batch = tvm.var(“input\_batch”) input\_channel = tvm.var(“input\_channel”) input\_width = tvm.var(“input\_width”) input\_height = tvm.var(“input\_height”) filter\_num = tvm.var(“filter\_num”) filter\_size = tvm.var(“filter\_size”)

> # Additional parameters # conv\_mode is always 1 and not used. I currently does not handle conv\_group. vertical\_pad = tvm.var(“vertical\_pad”) horizontal\_pad = tvm.var(“horizontal\_pad”) vertical\_stride = tvm.var(“vertical\_stride”) horizontal\_stride = tvm.var(“horizontal\_stride”)

> # Input data matrix and filter matrix. NCHW InputDataMatrix = tvm.placeholder((input\_batch, input\_channel, input\_height, input\_width), &nbsp;&nbsp;&nbsp;&nbsp;name=“InputDataMatrix”) FilterMatrix = tvm.placeholder((filter\_num, input\_channel, filter\_size, filter\_size), &nbsp;&nbsp;&nbsp;&nbsp;name=“FilterMatrix”)

> # Output dimensions output\_width = (input\_width - filter\_size + 2 \* horizontal\_pad) // horizontal\_stride + 1 output\_height = (input\_height - filter\_size + 2 \* vertical\_pad) // vertical\_stride + 1

> # Define padding InputDataMatrixPad = tvm.compute( &nbsp;&nbsp;&nbsp;&nbsp;(input\_batch, input\_channel, output\_height + 2 \* vertical\_pad, output\_width + 2 \* horizontal\_pad), &nbsp;&nbsp;&nbsp;&nbsp;lambda nidx, cidx, hidx, widx: tvm.if\_then\_else( &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;tvm.all(hidx - vertical\_pad \>= 0, hidx - vertical\_pad \< input\_height, &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;widx - horizontal\_pad \>= 0, widx - horizontal\_pad \< input\_width), &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;InputDataMatrix[nidx, cidx, hidx - vertical\_pad, widx - horizontal\_pad], tvm.const(0., “float32”) &nbsp;&nbsp;&nbsp;&nbsp;), name=“InputDataMatrixPad” )

> # Create reduction variables rc = tvm.reduce\_axis((0, input\_channel), name=‘rc’) rh = tvm.reduce\_axis((0, filter\_size), name=‘rh’) rw = tvm.reduce\_axis((0, filter\_size), name=‘rw’)

> # Compute the convolution OutputMatrix = tvm.compute( &nbsp;&nbsp;&nbsp;&nbsp;(input\_batch, filter\_num, output\_height, output\_width), &nbsp;&nbsp;&nbsp;&nbsp;lambda nidx, fidx, hidx, widx: tvm.sum( &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;InputDataMatrixPad[nidx, rc, hidx \* vertical\_stride + rh, widx \* horizontal\_stride + rw] \* &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;FilterMatrix[fidx, rc, rh, rw], &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;axis=[rc, rh, rw] &nbsp;&nbsp;&nbsp;&nbsp;), name=“OutputMatrix” )

> ######################## # Step 2. Design the memory hierarchy ########################

> # Schedule. Do padding inline. schedule = tvm.create\_schedule(OutputMatrix.op) schedule[InputDataMatrixPad].compute\_inline()

> # Memory read and write InputDataMatrixPadInShared = schedule.cache\_read(InputDataMatrixPad, ‘shared’, [OutputMatrix]) FilterMatrixInShared = schedule.cache\_read(FilterMatrix, “shared”, [OutputMatrix]) InputDataMatrixPadInLocal = schedule.cache\_read(InputDataMatrixPadInShared, “local”, &nbsp;&nbsp;&nbsp;&nbsp;[OutputMatrix]) FilterMatrixInLocal = schedule.cache\_read(FilterMatrixInShared, “local”, [OutputMatrix]) OutputMatrixInLocal = schedule.cache\_write(OutputMatrix, “local”)

> ######################## # Step 3. Blocking ########################

> # For each input image x num filter x image channel, a block is created. nidx, fidx, hidx, widx = schedule[OutputMatrix].op.axis schedule[OutputMatrix].bind(nidx, tvm.thread\_axis(“blockIdx.z”)) schedule[OutputMatrix].bind(fidx, tvm.thread\_axis(“blockIdx.y”)) rcidx, rhidx, rwidx = schedule[OutputMatrix].op.reduce\_axis schedule[OutputMatrix].bind(rcidx, tvm.thread\_axis(“blockIdx.x”))

> # Then, we are left with 2D convolution # Input[height \* weight] conv. Filter[size \* size] = Output[hidx \* widx] # We split the workload by a factor of 32 hwidxSplitFactor = 32 hwidxFused = schedule[OutputMatrix].fuse(hidx, widx) hwidxFusedSplit, \_ = schedule[OutputMatrix].split(hwidxFused, factor=hwidxSplitFactor) schedule[OutputMatrix].bind(hwidxFusedSplit, tvm.thread\_axis(“threadIdx.x”))

> ######################## # Step 4. Memory Fetching ########################

> schedule[OutputMatrixInLocal].compute\_at(schedule[OutputMatrix], hwidxFusedSplit) rcidx, rhidx, rwidx = schedule[OutputMatrixInLocal].op.reduce\_axis rhrwFused = schedule[OutputMatrixInLocal].fuse(rhidx, rwidx) schedule[InputDataMatrixPadInShared].compute\_at(schedule[OutputMatrixInLocal], rhrwFused) schedule[InputDataMatrixPadInLocal].compute\_at(schedule[OutputMatrixInLocal], rhrwFused) schedule[FilterMatrixInShared].compute\_at(schedule[OutputMatrixInLocal], rhrwFused) schedule[FilterMatrixInLocal].compute\_at(schedule[OutputMatrixInLocal], rhrwFused)

> ######################## # Step 5. Testing ########################

> func = tvm.build(schedule, [InputDataMatrix, FilterMatrix, vertical\_pad, horizontal\_pad, vertical\_stride, horizontal\_stride, OutputMatrix], ‘cuda’) ctx = tvm.gpu(0)

> input\_batch\_v = 2 input\_channel\_v = 3 input\_width\_v = 3 input\_height\_v = 3 filter\_num\_v = 3 filter\_size\_v = 3 vertical\_pad\_v = 1 horizontal\_pad\_v = 1 vertical\_stride\_v = 1 horizontal\_stride\_v = 1

> # Input a\_np = np.random.uniform(size=(input\_batch\_v, input\_channel\_v, input\_height\_v, &nbsp;&nbsp;&nbsp;&nbsp;input\_width\_v)).astype(InputDataMatrix.dtype) w\_np = np.random.uniform(size=(filter\_num\_v, input\_channel\_v, filter\_size\_v, &nbsp;&nbsp;&nbsp;&nbsp;filter\_size\_v)).astype(FilterMatrix.dtype) a = tvm.nd.array(a\_np, ctx) w = tvm.nd.array(w\_np, ctx)

> # Output output\_width\_v = (input\_width\_v - filter\_size\_v + 2 \* horizontal\_pad\_v) // horizontal\_stride\_v + 1 output\_height\_v = (input\_height\_v - filter\_size\_v + 2 \* vertical\_pad\_v) // vertical\_stride\_v + 1 b = tvm.nd.array(np.zeros((input\_batch\_v, filter\_num\_v, output\_height\_v, output\_width\_v), &nbsp;&nbsp;&nbsp;&nbsp;dtype=OutputMatrix.dtype), ctx)

> func(a, w, b, vertical\_pad\_v, horizontal\_pad\_v, vertical\_stride\_v, horizontal\_stride\_v) evaluator = func.time\_evaluator(func.entry\_name, ctx, number=1) print(‘Convolution: %f ms’ % (evaluator(a, w, b).mean \* 1e3))

**However, they don’t quite work. The first error I am getting is:**

> Traceback (most recent call last): File “matrixConvolution.py”, line 99, in rcidx, rhidx, rwidx = schedule[OutputMatrix].op.reduce\_axis ValueError: not enough values to unpack (expected 3, got 0)

**Please help me with the implementation. Also, if you know a better tutorial, either in TVM or CUDA, please share it with me. Thank you.**

---

<div class="post-metadata">

**Author:** ![comaniac](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/comaniac/32/899_2.png) [@comaniac](https://discuss.tvm.apache.org/u/comaniac)\
**Post date:** [October 20, 2019, 5:36am UTC](https://discuss.tvm.apache.org/t/implement-convolution-using-tvm/4393/2 "2019-10-20T05:36:25Z")

</div>

If all you need is to get conv2d working on TVM, you can directly use the `conv2d` op that has been defined and optimized. For example, the following code takes data (NCHW) and weight (OIHW), executes a conv2d with stride (2, 2), and produces output (NCHW). Since the target is set to `cuda`, it will automatically use the well-defined schedule for CUDA on GPU. If you would like to use cuBLAS/cuDNN, you can simply change the target to `cuda -libs=cublas,cudnn`.

```
import numpy as np
from tvm import relay
from tvm.contrib import graph_runtime

data = relay.var('data', shape=(1, 3, 224, 224))
weight = relay.var('weight', shape=(32, 3, 3, 3))
out = relay.nn.conv2d(data, weight, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
fun = relay.Function([data, weight], out)
mod = relay.Module()
mod['main'] = fun
target = tvm.target.cuda()
with relay.build_config(opt_level=3):
    graph, lib, params = relay.build_module.build(mod, target, params={})

ctx = tvm.gpu()
i_data = np.random.uniform(0, 1, size=(1, 3, 224, 224)).astype('float32')
i_weight= np.random.uniform(0, 1, size=(32, 3, 3, 3)).astype('float32')

module = graph_runtime.create(graph, lib, ctx)
module.set_input('data', i_data)
module.set_input('weight', i_weight)
module.run()
out = module.get_output(0, tvm.nd.empty((1, 32, 112, 112))).asnumpy()
```

---

<div class="post-metadata">

**Author:** ![ziangw2](https://discuss.tvm.apache.org/letter_avatar_proxy/v4/letter/z/3d9bf3/32.png) [@ziangw2](https://discuss.tvm.apache.org/u/ziangw2)\
**Post date:** [October 20, 2019, 5:40pm UTC](https://discuss.tvm.apache.org/t/implement-convolution-using-tvm/4393/5 "2019-10-20T17:40:31Z")

</div>

I am testing a new way of doing convolution, so I really need to have an implementation for that. TVM Tensor Expression hides a lot of complexity, which is great. I just can’t understand why I am getting that error. Why is the reduce\_axis member none?

---

<div class="post-metadata">

**Author:** ![hwu36](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/hwu36/32/2380_2.png) [@hwu36](https://discuss.tvm.apache.org/u/hwu36)\
**Post date:** [February 28, 2021, 4:17am UTC](https://discuss.tvm.apache.org/t/implement-convolution-using-tvm/4393/6 "2021-02-28T04:17:10Z")

</div>

> [@ziangw2](#):
>
> I tried to use cutlass but it is too complicated.

Cutlass open sourced NHWC 2D convolution. It is not difficult to change to use NCHW. Feel free to ask questions on cutlass github.
