# Autotuner fails for CNN in new dev branch **URL:** \ **Category:** Troubleshooting\ **Created:** [March 26, 2020, 12:00pm UTC](https://discuss.tvm.apache.org/t/autotuner-fails-for-cnn-in-new-dev-branch/6099 "2020-03-26T12:00:56Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Wheest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wheest/32/607_2.png) [@Wheest](https://discuss.tvm.apache.org/u/Wheest)\ **Post date:** [March 26, 2020, 12:00pm UTC](https://discuss.tvm.apache.org/t/autotuner-fails-for-cnn-in-new-dev-branch/6099/1 "2020-03-26T12:00:56Z")
I’ve just moved from `v0.6` to the development branch. [HEAD at time of writing](https://github.com/apache/incubator-tvm/tree/fdc8b0dd1763aece4ce457a7baf522c2989ac6c4). I’m trying to autotune one of my benchmarks, a CNN. It autotunes fine in v0.6, however with the dev branch, even with a couple of iterations it crashes with error: ```auto [XX:XX:XX] ../src/printer/doc.cc:55: text node: ' an internal invariant was violated while typechecking your program [XX:XX:XX] ../src/relay/op/nn/convolution.cc:561: Check failed: param->ker nel_size.defined() && param->channels.defined(): The kernel size and channels of a Conv must be set or infered by previous pass ``` and with other possibly relevant information: ```auto %15 = nn.contrib_conv2d_winograd_without_weight_transform(%11, %14, tile_size=4, padding=[1, 1, 1, 1], kernel_size=[3, 3]) an internal invariant was violated while typechecking your program [XX:XX:XX] ../src/relay/op/nn/convolution.cc:561: Check failed: param->kernel_size.defined() && param->channels.defined(): The kernel size and channels of a Conv must be set or infered by previous pass ``` The model runs fine when not autotuning. I’ve tried a couple of other models and they seem to work. The debug doesn’t mention any other layers failing, but it could be this is just the first to fail. My uninformed guess from the message is that somewhere along the way with `nn.contrib_conv2d_winograd_without_weight_transform` the kernel size or channels information is lost somewhere. However I’m not yet familiar enough with the C++ backend of tvm to know a good approach to start tracing. I can provide more information about my setup if needed. I have not jumped backwards through the commit history to see if there’s an earlier commit of this version that works. (off-topic, but does anyone have a recommended nice automated way of doing this beyond bash scripting, I guess doing some fort of binary search in a range of commits?) ---
**Author:** ![haichen](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/haichen/32/246_2.png) [@haichen](https://discuss.tvm.apache.org/u/haichen)\ **Post date:** [March 27, 2020, 4:27pm UTC](https://discuss.tvm.apache.org/t/autotuner-fails-for-cnn-in-new-dev-branch/6099/2 "2020-03-27T16:27:17Z")
Could you share the script that can reproduce the error? I think it’s likely because you didn’t specify the weight shape to depthwise conv2d op. ---
**Author:** ![Wheest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wheest/32/607_2.png) [@Wheest](https://discuss.tvm.apache.org/u/Wheest)\ **Post date:** [April 13, 2020, 5:57pm UTC](https://discuss.tvm.apache.org/t/autotuner-fails-for-cnn-in-new-dev-branch/6099/3 "2020-04-13T17:57:16Z")
An update with a reproducible example. Seems that it runs okay on x86 platforms, but on ARM CPU platforms I get the error described above. Have tried again with a more recent commit, and the same issue occurs. I’ve found it with two of my test models, which are in the repo as Onnx models. [See here](https://github.com/Wheest/tvm_autotuning_issue). Two models, ResNet34, and the smaller WRN-40-2. They don’t have any depthwise conv2d ops as far as I’m aware. You don’t need many trials to see the behaviour. A couple of example runs for the example are below: ```auto python tvm_autotune.py --target_type remote --device_key hikey --target_string 'llvm -device=arm_cpu -target=aarch64-linux-gnu' --output_path /tmp/ --host_port 9190 --trials 5 python tvm_autotune.py --target_type local --target_string 'llvm -target=x86_64-linux-gnu -mcpu=core-avx2' --output_path /tmp/ --trials 5 ``` ---
**Author:** ![haichen](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/haichen/32/246_2.png) [@haichen](https://discuss.tvm.apache.org/u/haichen)\ **Post date:** [May 6, 2020, 6:43pm UTC](https://discuss.tvm.apache.org/t/autotuner-fails-for-cnn-in-new-dev-branch/6099/4 "2020-05-06T18:43:53Z")
Could you try apply this [patch](https://github.com/apache/incubator-tvm/pull/5532) and see if it fixes the problem? ---
**Author:** ![Wheest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wheest/32/607_2.png) [@Wheest](https://discuss.tvm.apache.org/u/Wheest)\ **Post date:** [May 11, 2020, 3:47pm UTC](https://discuss.tvm.apache.org/t/autotuner-fails-for-cnn-in-new-dev-branch/6099/5 "2020-05-11T15:47:36Z")
Thanks for getting back to me on this. I’ve since updated to a newer commit (95e06b3). However, the autotuning on ARM CPU problem seems to have extended to all of my autotuning benchmarks. I’ve tried applying your patch to 95e06b3, but it does not seem to change the behavior. The autotuning process runs successfully. However, when evaluating the final model, I get the following error host side: ```auto Traceback (most recent call last): File "/home/[redacted]/tools/tvm/python/tvm/_ffi/_ctypes/ndarray.py", line 80, in __del__ check_call(_LIB.TVMArrayFree(self.handle)) File "/home/[redacted]/tools/tvm/python/tvm/_ffi/base.py", line 329, in check_call raise get_last_ffi_error() tvm._ffi.base.TVMError: Traceback (most recent call last): [bt] (7) /home/[redacted]/tools/tvm/build/libtvm.so(TVMArrayFree+0x23) [0x7fe4e0857263] [bt] (6) /home/[redacted]/tools/tvm/build/libtvm.so(tvm::runtime::RPCWrappedFunc::RemoteNDArrayDeleter(tvm::runtime::Object*)+0x1f) [0x7fe4e0887cbf] [bt] (5) /home/[redacted]/tools/tvm/build/libtvm.so(tvm::runtime::RPCClientSession::FreeHandle(void*, int)+0x7f) [0x7fe4e0881b1f] [bt] (4) /home/[redacted]/tools/tvm/build/libtvm.so(+0xd977c1) [0x7fe4e087c7c1] [bt] (3) /home/[redacted]/tools/tvm/build/libtvm.so(tvm::runtime::RPCEndpoint::HandleUntilReturnEvent(bool, std::function)+0x177) [0x7fe4e0879fe7] [bt] (2) /home/[redacted]/tools/tvm/build/libtvm.so(tvm::runtime::SockChannel::Send(void const*, unsigned long)+0x20) [0x7fe4e088f010] [bt] (1) /home/[redacted]/tools/tvm/build/libtvm.so(tvm::support::Socket::Error(char const*)+0xe6) [0x7fe4e088eef6] [bt] (0) /home/[redacted]/tools/tvm/build/libtvm.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x32) [0x7fe4dfeff472] File "../src/runtime/rpc/../../support/socket.h", line 362 TVMError: Socket SockChannel::Send Error:Broken pipe ``` On the device side, I get the error: ```auto malloc(): invalid size (unsorted) ``` I have tested this on several ARM devices, and network benchmarks. The autotuner works on the x86 platform running locally. Searching the forum, the only related issue I can find is [this one](https://discuss.tvm.ai/t/tvmerror-check-failed-code-rpccode-code-4/4878), however no resolution. ---
**Author:** ![Wheest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wheest/32/607_2.png) [@Wheest](https://discuss.tvm.apache.org/u/Wheest)\ **Post date:** [May 11, 2020, 4:01pm UTC](https://discuss.tvm.apache.org/t/autotuner-fails-for-cnn-in-new-dev-branch/6099/6 "2020-05-11T16:01:41Z")
For completeness, I have also tried patch on the original commit in this issue ([fdc8b0dd1763](https://github.com/apache/incubator-tvm/tree/fdc8b0dd1763aece4ce457a7baf522c2989ac6c4)). I confirm that in this case, yes the patch works. It seems that I have inadvertently walked into another issue with the newer commit. Should I keep this thread for that issue, or create a new one? ---
**Author:** ![haichen](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/haichen/32/246_2.png) [@haichen](https://discuss.tvm.apache.org/u/haichen)\ **Post date:** [May 11, 2020, 4:16pm UTC](https://discuss.tvm.apache.org/t/autotuner-fails-for-cnn-in-new-dev-branch/6099/7 "2020-05-11T16:16:53Z")
Did you update the TVM on the device side as well? It might because the newer version is not compatible with old ones. ---
**Author:** ![Wheest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wheest/32/607_2.png) [@Wheest](https://discuss.tvm.apache.org/u/Wheest)\ **Post date:** [May 11, 2020, 4:19pm UTC](https://discuss.tvm.apache.org/t/autotuner-fails-for-cnn-in-new-dev-branch/6099/8 "2020-05-11T16:19:09Z")
Good check, yes. Should have clarified I had done this. I ensure that both device and host sides are compiled on the same commit (I’ve [learned that lesson](https://discuss.tvm.ai/t/solved-autotvm-runtimeerror-return-code-4-during-autotuning/1010/5)). ---
**Author:** ![Wheest](https://discuss.tvm.apache.org/user_avatar/discuss.tvm.apache.org/wheest/32/607_2.png) [@Wheest](https://discuss.tvm.apache.org/u/Wheest)\ **Post date:** [May 11, 2020, 4:28pm UTC](https://discuss.tvm.apache.org/t/autotuner-fails-for-cnn-in-new-dev-branch/6099/9 "2020-05-11T16:28:50Z")
I’ve now applied the patch to an even more recent commit (0e877521f). I guess there was some error introduced somewhere leading up to and including 95e06b3 which was later corrected. So to confirm, your patch works, but only on earlier and later versions of the repo which do not have other issues. I’ve not delved too deeply into TVM testing infrastructure. Would this have been caught running a CI autotuning test on an ARM instance for a conv net? Are those currently covered, if so could you point me towards them?