AutoTVM fails with "Could not find any valid schedule for task Task(func_name=conv2d_NCHWc.x86...""

Hi, I’m trying to tune a simple CNN with inverted bottleneck layers. This is the command I’m using:

tvmc tune --target "llvm -mcpu=skylake-avx512" --output /tmp/dummy.json --number 250 inverted_bottleneck_channels=312_resolution=38_quantization_mode=off.onnx

And here is the .onnx model file I’m trying to tune.

There is a lot of output, but this seems to be the relevant segment:

WARNING:autotvm:Too many errors happen in the tuning. Switching to debug mode.
DEBUG:autotvm:No: 331   GFLOPS: 0.00/0.00       result: MeasureResult(costs=(RuntimeError('Traceback (most
 recent call last):\n  78: 0xffffffffffffffff\n  77: _start\n  76: __libc_start_main\n        at ../csu/li
bc-start.c:308\n  75: Py_BytesMain\n        at Modules/main.c:743\n  74: Py_RunMain\n        at Modules/ma
in.c:689\n  73: pymain_run_python\n        at Modules/main.c:604\n  72: pymain_run_module\n        at Modu
les/main.c:303\n  71: PyVectorcall_Call\n        at Objects/call.c:200\n  70: _PyFunction_Vectorcall\n
    at Objects/call.c:436\n  69: _PyEval_EvalCodeWithName\n        at Python/ceval.c:4298\n  68: _PyEval_E
valFrameDefault\n        at Python/ceval.c:3500\n  67: call_function\n        at Python/ceval.c:4963\n  66
: _PyObject_Vectorcall\n        at ./Include/cpython/abstract.h:127\n  65: _PyFunction_Vectorcall\n
 at Objects/call.c:436\n  64: _PyEval_EvalCodeWithName\n        at Python/ceval.c:4298\n  63: _PyEval_Eval
FrameDefault\n        at Python/ceval.c:3500\n  62: call_function\n        at Python/ceval.c:4963\n  61: _
PyObject_Vectorcall\n        at ./Include/cpython/abstract.h:127\n  60: cfunction_vectorca'),), error_no=M
easureErrorNo.RUNTIME_DEVICE, all_cost=13.823271989822388, timestamp=1656278865.1227567)  [('tile_ic', [-1
, 156]), ('tile_oc', [-1, 156]), ('tile_ow', [-1, 4]), ('tile_oh', 1)],None,1806
DEBUG:autotvm:No: 332   GFLOPS: 0.00/0.00       result: MeasureResult(costs=(RuntimeError('Traceback (most
 recent call last):\n  78: 0xffffffffffffffff\n  77: _start\n  76: __libc_start_main\n        at ../csu/li
bc-start.c:308\n  75: Py_BytesMain\n        at Modules/main.c:743\n  74: Py_RunMain\n        at Modules/ma
in.c:689\n  73: pymain_run_python\n        at Modules/main.c:604\n  72: pymain_run_module\n        at Modu
les/main.c:303\n  71: PyVectorcall_Call\n        at Objects/call.c:200\n  70: _PyFunction_Vectorcall\n
    at Objects/call.c:436\n  69: _PyEval_EvalCodeWithName\n        at Python/ceval.c:4298\n  68: _PyEval_E
valFrameDefault\n        at Python/ceval.c:3500\n  67: call_function\n        at Python/ceval.c:4963\n  66
: _PyObject_Vectorcall\n        at ./Include/cpython/abstract.h:127\n  65: _PyFunction_Vectorcall\n
 at Objects/call.c:436\n  64: _PyEval_EvalCodeWithName\n        at Python/ceval.c:4298\n  63: _PyEval_Eval
FrameDefault\n        at Python/ceval.c:3500\n  62: call_function\n        at Python/ceval.c:4963\n  61: _
PyObject_Vectorcall\n        at ./Include/cpython/abstract.h:127\n  60: cfunction_vectorca'),), error_no=M
easureErrorNo.RUNTIME_DEVICE, all_cost=10.371805429458618, timestamp=1656278875.2470813)  [('tile_ic', [-1
, 39]), ('tile_oc', [-1, 3]), ('tile_ow', [-1, 38]), ('tile_oh', 1)],None,1962
WARNING:autotvm:Too many errors happen in the tuning. Switching to debug mode.
DEBUG:autotvm:No: 333   GFLOPS: 0.00/0.00       result: MeasureResult(costs=(RuntimeError('Traceback (most
 recent call last):\n  78: 0xffffffffffffffff\n  77: _start\n  76: __libc_start_main\n        at ../csu/li
bc-start.c:308\n  75: Py_BytesMain\n        at Modules/main.c:743\n  74: Py_RunMain\n        at Modules/ma
in.c:689\n  73: pymain_run_python\n        at Modules/main.c:604\n  72: pymain_run_module\n        at Modu
les/main.c:303\n  71: PyVectorcall_Call\n        at Objects/call.c:200\n  70: _PyFunction_Vectorcall\n
    at Objects/call.c:436\n  69: _PyEval_EvalCodeWithName\n        at Python/ceval.c:4298\n  68: _PyEval_E
valFrameDefault\n        at Python/ceval.c:3500\n  67: call_function\n        at Python/ceval.c:4963\n  66
: _PyObject_Vectorcall\n        at ./Include/cpython/abstract.h:127\n  65: _PyFunction_Vectorcall\n
 at Objects/call.c:436\n  64: _PyEval_EvalCodeWithName\n        at Python/ceval.c:4298\n  63: _PyEval_Eval
FrameDefault\n        at Python/ceval.c:3500\n  62: call_function\n        at Python/ceval.c:4963\n  61: _
PyObject_Vectorcall\n        at ./Include/cpython/abstract.h:127\n  60: cfunction_vectorca'),), error_no=M
easureErrorNo.RUNTIME_DEVICE, all_cost=11.086325645446777, timestamp=1656278886.4471989)  [('tile_ic', [-1
, 78]), ('tile_oc', [-1, 36]), ('tile_ow', [-1, 38]), ('tile_oh', 1)],None,2140
DEBUG:autotvm:Early stopped. Best iter: 0.
WARNING:root:Could not find any valid schedule for task Task(func_name=conv2d_NCHWc.x86, args=(('TENSOR',
(1, 312, 38, 38), 'float32'), ('TENSOR', (1872, 312, 1, 1), 'float32'), (1, 1), (0, 0, 0, 0), (1, 1), 'NCH
W', 'NCHW', 'float32'), kwargs={}, workload=('conv2d_NCHWc.x86', ('TENSOR', (1, 312, 38, 38), 'float32'),
('TENSOR', (1872, 312, 1, 1), 'float32'), (1, 1), (0, 0, 0, 0), (1, 1), 'NCHW', 'NCHW', 'float32')). A fil
e containing the errors has been written to /tmp/tvm_tuning_errors_4x0wg3cv.log.
INFO:autotvm:Get devices for measurement successfully!

I’ve also uploaded the full output here.

  • TVM 0.8.0
  • Python 3.8.12
  • xgboost 1.5.0
  • OS: Debian GNU/Linux 11 (bullseye)
  • /proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) CPU @ 2.00GHz
stepping	: 3
microcode	: 0x1
cpu MHz		: 1999.999
cache size	: 39424 KB
physical id	: 0
siblings	: 2
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat md_clear arch_capabilities
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa
bogomips	: 3999.99
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

and one more identical processor.

Any hints as to what the issue might be? I encountered this with other CNN models (different sizes, some dilations) as well. Thanks!

Hi @vvolhejn ,

I can talk from my experience: that error “Could not find any valid schedule for task…” means there was an error during the build process of the different schedules. This could happen for a lot of reasons (strange tiling factors, problems with the compute_at methods in the schedule, knobs without valid options, double buffering set in incorrect loops, etc).

I have no experience with that target, but I stumbled accross this kind of errors when I was adding my own target.

It seems to me there is an incompatibility between the parameters of your convolutional layer and how the schedule for that layer was defined, but I would really ask someone more experienced than me about this.