Hi, I’m trying to tune a simple CNN with inverted bottleneck layers. This is the command I’m using:
tvmc tune --target "llvm -mcpu=skylake-avx512" --output /tmp/dummy.json --number 250 inverted_bottleneck_channels=312_resolution=38_quantization_mode=off.onnx
And here is the .onnx model file I’m trying to tune.
There is a lot of output, but this seems to be the relevant segment:
WARNING:autotvm:Too many errors happen in the tuning. Switching to debug mode.
DEBUG:autotvm:No: 331 GFLOPS: 0.00/0.00 result: MeasureResult(costs=(RuntimeError('Traceback (most
recent call last):\n 78: 0xffffffffffffffff\n 77: _start\n 76: __libc_start_main\n at ../csu/li
bc-start.c:308\n 75: Py_BytesMain\n at Modules/main.c:743\n 74: Py_RunMain\n at Modules/ma
in.c:689\n 73: pymain_run_python\n at Modules/main.c:604\n 72: pymain_run_module\n at Modu
les/main.c:303\n 71: PyVectorcall_Call\n at Objects/call.c:200\n 70: _PyFunction_Vectorcall\n
at Objects/call.c:436\n 69: _PyEval_EvalCodeWithName\n at Python/ceval.c:4298\n 68: _PyEval_E
valFrameDefault\n at Python/ceval.c:3500\n 67: call_function\n at Python/ceval.c:4963\n 66
: _PyObject_Vectorcall\n at ./Include/cpython/abstract.h:127\n 65: _PyFunction_Vectorcall\n
at Objects/call.c:436\n 64: _PyEval_EvalCodeWithName\n at Python/ceval.c:4298\n 63: _PyEval_Eval
FrameDefault\n at Python/ceval.c:3500\n 62: call_function\n at Python/ceval.c:4963\n 61: _
PyObject_Vectorcall\n at ./Include/cpython/abstract.h:127\n 60: cfunction_vectorca'),), error_no=M
easureErrorNo.RUNTIME_DEVICE, all_cost=13.823271989822388, timestamp=1656278865.1227567) [('tile_ic', [-1
, 156]), ('tile_oc', [-1, 156]), ('tile_ow', [-1, 4]), ('tile_oh', 1)],None,1806
DEBUG:autotvm:No: 332 GFLOPS: 0.00/0.00 result: MeasureResult(costs=(RuntimeError('Traceback (most
recent call last):\n 78: 0xffffffffffffffff\n 77: _start\n 76: __libc_start_main\n at ../csu/li
bc-start.c:308\n 75: Py_BytesMain\n at Modules/main.c:743\n 74: Py_RunMain\n at Modules/ma
in.c:689\n 73: pymain_run_python\n at Modules/main.c:604\n 72: pymain_run_module\n at Modu
les/main.c:303\n 71: PyVectorcall_Call\n at Objects/call.c:200\n 70: _PyFunction_Vectorcall\n
at Objects/call.c:436\n 69: _PyEval_EvalCodeWithName\n at Python/ceval.c:4298\n 68: _PyEval_E
valFrameDefault\n at Python/ceval.c:3500\n 67: call_function\n at Python/ceval.c:4963\n 66
: _PyObject_Vectorcall\n at ./Include/cpython/abstract.h:127\n 65: _PyFunction_Vectorcall\n
at Objects/call.c:436\n 64: _PyEval_EvalCodeWithName\n at Python/ceval.c:4298\n 63: _PyEval_Eval
FrameDefault\n at Python/ceval.c:3500\n 62: call_function\n at Python/ceval.c:4963\n 61: _
PyObject_Vectorcall\n at ./Include/cpython/abstract.h:127\n 60: cfunction_vectorca'),), error_no=M
easureErrorNo.RUNTIME_DEVICE, all_cost=10.371805429458618, timestamp=1656278875.2470813) [('tile_ic', [-1
, 39]), ('tile_oc', [-1, 3]), ('tile_ow', [-1, 38]), ('tile_oh', 1)],None,1962
WARNING:autotvm:Too many errors happen in the tuning. Switching to debug mode.
DEBUG:autotvm:No: 333 GFLOPS: 0.00/0.00 result: MeasureResult(costs=(RuntimeError('Traceback (most
recent call last):\n 78: 0xffffffffffffffff\n 77: _start\n 76: __libc_start_main\n at ../csu/li
bc-start.c:308\n 75: Py_BytesMain\n at Modules/main.c:743\n 74: Py_RunMain\n at Modules/ma
in.c:689\n 73: pymain_run_python\n at Modules/main.c:604\n 72: pymain_run_module\n at Modu
les/main.c:303\n 71: PyVectorcall_Call\n at Objects/call.c:200\n 70: _PyFunction_Vectorcall\n
at Objects/call.c:436\n 69: _PyEval_EvalCodeWithName\n at Python/ceval.c:4298\n 68: _PyEval_E
valFrameDefault\n at Python/ceval.c:3500\n 67: call_function\n at Python/ceval.c:4963\n 66
: _PyObject_Vectorcall\n at ./Include/cpython/abstract.h:127\n 65: _PyFunction_Vectorcall\n
at Objects/call.c:436\n 64: _PyEval_EvalCodeWithName\n at Python/ceval.c:4298\n 63: _PyEval_Eval
FrameDefault\n at Python/ceval.c:3500\n 62: call_function\n at Python/ceval.c:4963\n 61: _
PyObject_Vectorcall\n at ./Include/cpython/abstract.h:127\n 60: cfunction_vectorca'),), error_no=M
easureErrorNo.RUNTIME_DEVICE, all_cost=11.086325645446777, timestamp=1656278886.4471989) [('tile_ic', [-1
, 78]), ('tile_oc', [-1, 36]), ('tile_ow', [-1, 38]), ('tile_oh', 1)],None,2140
DEBUG:autotvm:Early stopped. Best iter: 0.
WARNING:root:Could not find any valid schedule for task Task(func_name=conv2d_NCHWc.x86, args=(('TENSOR',
(1, 312, 38, 38), 'float32'), ('TENSOR', (1872, 312, 1, 1), 'float32'), (1, 1), (0, 0, 0, 0), (1, 1), 'NCH
W', 'NCHW', 'float32'), kwargs={}, workload=('conv2d_NCHWc.x86', ('TENSOR', (1, 312, 38, 38), 'float32'),
('TENSOR', (1872, 312, 1, 1), 'float32'), (1, 1), (0, 0, 0, 0), (1, 1), 'NCHW', 'NCHW', 'float32')). A fil
e containing the errors has been written to /tmp/tvm_tuning_errors_4x0wg3cv.log.
INFO:autotvm:Get devices for measurement successfully!
I’ve also uploaded the full output here.
- TVM 0.8.0
- Python 3.8.12
- xgboost 1.5.0
- OS: Debian GNU/Linux 11 (bullseye)
- /proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 85
model name : Intel(R) Xeon(R) CPU @ 2.00GHz
stepping : 3
microcode : 0x1
cpu MHz : 1999.999
cache size : 39424 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat md_clear arch_capabilities
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa
bogomips : 3999.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
and one more identical processor.
Any hints as to what the issue might be? I encountered this with other CNN models (different sizes, some dilations) as well. Thanks!