# \[autoscheduler\] GPU failure on compile for BERT-like models

**URL:** <https://discuss.tvm.apache.org/t/autoscheduler-gpu-failure-on-compile-for-bert-like-models/10386>\
**Category:** Troubleshooting\
**Created:** [July 3, 2021, 5:58pm UTC](https://discuss.tvm.apache.org/t/autoscheduler-gpu-failure-on-compile-for-bert-like-models/10386 "2021-07-03T17:58:53Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wheest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wheest/32/607_2.png) [@Wheest](https://discuss.tvm.apache.org/u/Wheest)\
**Post date:** [July 3, 2021, 5:58pm UTC](https://discuss.tvm.apache.org/t/autoscheduler-gpu-failure-on-compile-for-bert-like-models/10386/1 "2021-07-03T17:58:53Z")

</div>

I’m trying to auto-schedule BERT-like models on a GPU. This works successfully on the CPU.

However things fail when running on the GPU. I successfully complete all of the tuning tasks, however when compiling the full model with the logfile, I always crash with the error:

```auto
File "/home/wheest/tools/tvm/python/tvm/relay/op/strategy/generic.py", line 767, in _compute_batch_matmul
  return [topi_compute(*args)]
File "/home/wheest/tools/tvm/python/tvm/autotvm/task/topi_integration.py", line 165, in wrapper
  node = topi_compute(cfg, *args)
File "/home/wheest/tools/tvm/python/tvm/topi/cuda/batch_matmul.py", line 32, in batch_matmul
  return nn.batch_matmul(x, y)
File "/home/wheest/tools/tvm/python/tvm/topi/nn/batch_matmul.py", line 57, in batch_matmul
    assert len(x_shape) == 3 and len(y_shape) == 3, "only support 3-dim batch_matmul"

```

Presumably Ansor is trying to do some sort of tiling and it is causing problems.

Is there an auto-scheduling flag or similar which can ensure we don’t break this batch\_matmul rule?

Or some other clever trick?

---

<div class="post-metadata">

**Author:** ![masahi](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/masahi/32/74_2.png) [@masahi](https://discuss.tvm.apache.org/u/masahi)\
**Post date:** [July 4, 2021, 1:46am UTC](https://discuss.tvm.apache.org/t/autoscheduler-gpu-failure-on-compile-for-bert-like-models/10386/2 "2021-07-04T01:46:47Z")

</div>

Hmm this is strange. I’ve done a lot of auto scheduling on transformer based models and didn’t encounter issues. cc @comaniac

---

<div class="post-metadata">

**Author:** ![comaniac](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/comaniac/32/899_2.png) [@comaniac](https://discuss.tvm.apache.org/u/comaniac)\
**Post date:** [July 4, 2021, 2:05am UTC](https://discuss.tvm.apache.org/t/autoscheduler-gpu-failure-on-compile-for-bert-like-models/10386/3 "2021-07-04T02:05:38Z")

</div>

Maybe something goes wrong in AutoScheduler when rewriting the layout. You can try to disable the layout rewritten in AutoScheduler by specifying disabled\_pass={“AutoSchedulerLayoutRewrite”} in PassContext.

Also cc @jcf94

---

<div class="post-metadata">

**Author:** ![Wheest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wheest/32/607_2.png) [@Wheest](https://discuss.tvm.apache.org/u/Wheest)\
**Post date:** [July 9, 2021, 6:03pm UTC](https://discuss.tvm.apache.org/t/autoscheduler-gpu-failure-on-compile-for-bert-like-models/10386/4 "2021-07-09T18:03:42Z")

</div>

Thanks for the help, though with this approach I got the classic computation outside GPU loop bound error:

```auto
 Did you forget to bind?
    Variable `placeholder` is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
    Variable `placeholder` is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
    Variable `T_dense` is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
    Variable `T_dense` is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
    Variable `T_dense` is directly accessed by host memory (it is not contained in a thread environment or in the function arguments.
  File "../src/tir/analysis/verify_memory.cc", line 202
RuntimeError: Memory verification failed with the following errors:
PrimFunc([placeholder, placeholder, T_dense]) attrs={"global_symbol": "fused_nn_dense_73", "tir.noalias": (bool)1, "target": opencl -keys=mali,opencl,gpu -device=mali -max_num_threads=256 -th
read_warp_size=1} {
  T_dense[0] = 0f
  for (k, 0, 768) {
    T_dense[0] = (T_dense[0] + (placeholder[k]*placeholder[k]))
  }
}

```

---

<div class="post-metadata">

**Author:** ![hope51607](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/hope51607/32/2898_2.png) [@hope51607](https://discuss.tvm.apache.org/u/hope51607)\
**Post date:** [August 3, 2021, 6:47am UTC](https://discuss.tvm.apache.org/t/autoscheduler-gpu-failure-on-compile-for-bert-like-models/10386/5 "2021-08-03T06:47:08Z")

</div>

> [@Wheest](#):
>
> I’m trying to auto-schedule BERT-like models on a GPU. This works successfully on the CPU.
> 
> However things fail when running on the GPU. I successfully complete all of the tuning tasks, however when compiling the full model with the logfile, I always crash with the error:
> 
> ```auto
> File "/home/wheest/tools/tvm/python/tvm/relay/op/strategy/generic.py", line 767, in _compute_batch_matmul
> return [topi_compute(*args)]
> File "/home/wheest/tools/tvm/python/tvm/autotvm/task/topi_integration.py", line 165, in wrapper
> node = topi_compute(cfg, *args)
> File "/home/wheest/tools/tvm/python/tvm/topi/cuda/batch_matmul.py", line 32, in batch_matmul
> return nn.batch_matmul(x, y)
> File "/home/wheest/tools/tvm/python/tvm/topi/nn/batch_matmul.py", line 57, in batch_matmul
> assert len(x_shape) == 3 and len(y_shape) == 3, "only support 3-dim batch_matmul"
> 
> ```

Hi I have encountered the same problem. Do you solved this problem yet?

---

<div class="post-metadata">

**Author:** ![Wheest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wheest/32/607_2.png) [@Wheest](https://discuss.tvm.apache.org/u/Wheest)\
**Post date:** [August 3, 2021, 10:01am UTC](https://discuss.tvm.apache.org/t/autoscheduler-gpu-failure-on-compile-for-bert-like-models/10386/6 "2021-08-03T10:01:27Z")

</div>

Hi @hope51607 unfortunately I have not found a solution to this problem yet.

---

<div class="post-metadata">

**Author:** ![hope51607](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/hope51607/32/2898_2.png) [@hope51607](https://discuss.tvm.apache.org/u/hope51607)\
**Post date:** [August 6, 2021, 6:03am UTC](https://discuss.tvm.apache.org/t/autoscheduler-gpu-failure-on-compile-for-bert-like-models/10386/7 "2021-08-06T06:03:04Z")

</div>

@Wheest I solved this problem by adding the following code in `python/tvm/relay/op/strategy/mali.py`

```python
@batch_matmul_strategy.register("mali")
def batch_matmul_strategy_mali(attrs, inputs, out_type, target):
    """batch_matmul mali strategy"""
    strategy = _op.OpStrategy()
    if not is_auto_scheduler_enabled():
        strategy.add_implementation(
            wrap_compute_batch_matmul(topi.nn.batch_matmul),
            wrap_topi_schedule(topi.nn.batch_matmul),
            name="batch_matmul.mali",
        )
    else:
        strategy.add_implementation(
            wrap_compute_batch_matmul(topi.nn.batch_matmul, need_auto_scheduler_layout=True),
            naive_schedule,
            name="batch_matmul.mali",
        )
    return strategy

```

I don’t know what target you used. But maybe it also works for you.