[External Codegen] DNNL Test Crashing

jszaday · August 2, 2021, 5:03pm

I have built TVM with LLVM and DNNL enabled, although I have not enabled MKL/MKLDNN/MKL BLAS. Possibly due to this “incomplete” set of build options, I am running into a crash when trying to run the following unit test:

tests/python/relay ‹main*› % python3 ./test_external_codegen.py
./test_external_codegen.py:68: DeprecationWarning: legacy graph executor behavior of producing json / lib / params will be removed in the next release. Please see documents of tvm.contrib.graph_executor.GraphModule for the  new recommended usage.
  json, lib, _ = relay.build(mod, target=target)
./test_external_codegen.py:68: DeprecationWarning: legacy graph executor behavior of producing json / lib / params will be removed in the next release. Please see documents of tvm.contrib.graph_executor.GraphModule for the  new recommended usage.
  json, lib, _ = relay.build(mod, target=target)
./test_external_codegen.py:68: DeprecationWarning: legacy graph executor behavior of producing json / lib / params will be removed in the next release. Please see documents of tvm.contrib.graph_executor.GraphModule for the  new recommended usage.
  json, lib, _ = relay.build(mod, target=target)
./test_external_codegen.py:68: DeprecationWarning: legacy graph executor behavior of producing json / lib / params will be removed in the next release. Please see documents of tvm.contrib.graph_executor.GraphModule for the  new recommended usage.
  json, lib, _ = relay.build(mod, target=target)
./test_external_codegen.py:253: DeprecationWarning: legacy graph executor behavior of producing json / lib / params will be removed in the next release. Please see documents of tvm.contrib.graph_executor.GraphModule for the  new recommended usage.
  _, _, params = relay.build(mod, target="llvm")
One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.
Traceback (most recent call last):
  File "./test_external_codegen.py", line 368, in <module>
    test_extern_dnnl()
  File "./test_external_codegen.py", line 296, in test_extern_dnnl
    mod, {"data0": i_data, "weight0": w_data}, (1, 32, 14, 14), ref_res.numpy(), tol=1e-5
  File "./test_external_codegen.py", line 80, in check_result
    check_vm_result()
  File "./test_external_codegen.py", line 62, in check_vm_result
    vm = runtime.vm.VirtualMachine(exe, device)
  File "/home/szaday2/workspace/tvm/python/tvm/runtime/vm.py", line 341, in __init__
    self.module = exe.mod["vm_load_executable"]()
  File "/home/szaday2/workspace/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 237, in __call__
    raise get_last_ffi_error()
tvm._ffi.base.TVMError: TVMError: could not construct a memory descriptor using a format tag

Do you have any suggestions on what to try to correct this failure? Do I need to enable MKL/etc. to use the DNNL target? If so, how do enable them? When I tried to build with them, I kept getting errors pertaining to USE_MKL pointing to the wrong directory, and MKLBLAS being missing.

Also, a secondary question – but assuming I get this all working, does the DNNL target support Relay QNN?

juierror · August 6, 2021, 2:06pm

I also have this error, but I already build with USE_MKL, USE_MKLDNN.

SergeS · August 10, 2021, 4:14am

Could you please provide your CMake log?

juierror · August 10, 2021, 7:22am

-- Build compiler with Relay VM profiler support...
-- Build with RPC support...
-- Build with Graph runtime support...
-- Build with Graph runtime debug support...
-- Build with Relay VM profiler support...
-- VTA build with VTA_HW_PATH=/home/siwa/tvm_official/3rdparty/vta-hw
-- Build VTA runtime with target: sim
-- Found CUDA_TOOLKIT_ROOT_DIR=/usr/local/cuda-10.1
-- Found CUDA_CUDA_LIBRARY=/usr/local/cuda-10.1/targets/x86_64-linux/lib/stubs/libcuda.so
-- Found CUDA_CUDART_LIBRARY=/usr/local/cuda-10.1/lib64/libcudart.so
-- Found CUDA_NVRTC_LIBRARY=/usr/local/cuda-10.1/lib64/libnvrtc.so
-- Found CUDA_CUDNN_LIBRARY=/usr/lib/x86_64-linux-gnu/libcudnn.so
-- Found CUDA_CUBLAS_LIBRARY=/usr/lib/x86_64-linux-gnu/libcublas.so
-- Found CUDA_CUBLASLT_LIBRARY=/usr/lib/x86_64-linux-gnu/libcublasLt.so
-- Build with CUDA support
-- Build with cuDNN support
-- Build with OpenMP /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.so;/usr/lib/x86_64-linux-gnu/libpthread.so
-- Use llvm-config=/home/siwa/llvm-10.0/bin/llvm-config
-- Found LLVM_INCLUDE_DIRS=/home/siwa/llvm-10.0/include
-- Found LLVM_DEFINITIONS=-D_GNU_SOURCE;-D__STDC_CONSTANT_MACROS;-D__STDC_FORMAT_MACROS;-D__STDC_LIMIT_MACROS
-- Found LLVM_LIBS=/home/siwa/llvm-10.0/lib/libLLVMXRay.a;/home/siwa/llvm-10.0/lib/libLLVMWindowsManifest.a;/home/siwa/llvm-10.0/lib/libLLVMTableGen.a;/home/siwa/llvm-10.0/lib/libLLVMSymbolize.a;/home/siwa/llvm-10.0/lib/libLLVMDebugInfoPDB.a;/home/siwa/llvm-10.0/lib/libLLVMOrcJIT.a;/home/siwa/llvm-10.0/lib/libLLVMOrcError.a;/home/siwa/llvm-10.0/lib/libLLVMJITLink.a;/home/siwa/llvm-10.0/lib/libLLVMObjectYAML.a;/home/siwa/llvm-10.0/lib/libLLVMMCA.a;/home/siwa/llvm-10.0/lib/libLLVMLTO.a;/home/siwa/llvm-10.0/lib/libLLVMPasses.a;/home/siwa/llvm-10.0/lib/libLLVMObjCARCOpts.a;/home/siwa/llvm-10.0/lib/libLLVMLineEditor.a;/home/siwa/llvm-10.0/lib/libLLVMLibDriver.a;/home/siwa/llvm-10.0/lib/libLLVMInterpreter.a;/home/siwa/llvm-10.0/lib/libLLVMFuzzMutate.a;/home/siwa/llvm-10.0/lib/libLLVMFrontendOpenMP.a;/home/siwa/llvm-10.0/lib/libLLVMMCJIT.a;/home/siwa/llvm-10.0/lib/libLLVMExecutionEngine.a;/home/siwa/llvm-10.0/lib/libLLVMRuntimeDyld.a;/home/siwa/llvm-10.0/lib/libLLVMDWARFLinker.a;/home/siwa/llvm-10.0/lib/libLLVMDlltoolDriver.a;/home/siwa/llvm-10.0/lib/libLLVMOption.a;/home/siwa/llvm-10.0/lib/libLLVMDebugInfoGSYM.a;/home/siwa/llvm-10.0/lib/libLLVMCoverage.a;/home/siwa/llvm-10.0/lib/libLLVMCoroutines.a;/home/siwa/llvm-10.0/lib/libLLVMXCoreDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMXCoreCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMXCoreDesc.a;/home/siwa/llvm-10.0/lib/libLLVMXCoreInfo.a;/home/siwa/llvm-10.0/lib/libLLVMX86Disassembler.a;/home/siwa/llvm-10.0/lib/libLLVMX86AsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMX86CodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMX86Desc.a;/home/siwa/llvm-10.0/lib/libLLVMX86Utils.a;/home/siwa/llvm-10.0/lib/libLLVMX86Info.a;/home/siwa/llvm-10.0/lib/libLLVMWebAssemblyDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMWebAssemblyCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMWebAssemblyDesc.a;/home/siwa/llvm-10.0/lib/libLLVMWebAssemblyAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMWebAssemblyInfo.a;/home/siwa/llvm-10.0/lib/libLLVMSystemZDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMSystemZCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMSystemZAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMSystemZDesc.a;/home/siwa/llvm-10.0/lib/libLLVMSystemZInfo.a;/home/siwa/llvm-10.0/lib/libLLVMSparcDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMSparcCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMSparcAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMSparcDesc.a;/home/siwa/llvm-10.0/lib/libLLVMSparcInfo.a;/home/siwa/llvm-10.0/lib/libLLVMRISCVDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMRISCVCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMRISCVAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMRISCVDesc.a;/home/siwa/llvm-10.0/lib/libLLVMRISCVUtils.a;/home/siwa/llvm-10.0/lib/libLLVMRISCVInfo.a;/home/siwa/llvm-10.0/lib/libLLVMPowerPCDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMPowerPCCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMPowerPCAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMPowerPCDesc.a;/home/siwa/llvm-10.0/lib/libLLVMPowerPCInfo.a;/home/siwa/llvm-10.0/lib/libLLVMNVPTXCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMNVPTXDesc.a;/home/siwa/llvm-10.0/lib/libLLVMNVPTXInfo.a;/home/siwa/llvm-10.0/lib/libLLVMMSP430Disassembler.a;/home/siwa/llvm-10.0/lib/libLLVMMSP430CodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMMSP430AsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMMSP430Desc.a;/home/siwa/llvm-10.0/lib/libLLVMMSP430Info.a;/home/siwa/llvm-10.0/lib/libLLVMMipsDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMMipsCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMMipsAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMMipsDesc.a;/home/siwa/llvm-10.0/lib/libLLVMMipsInfo.a;/home/siwa/llvm-10.0/lib/libLLVMLanaiDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMLanaiCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMLanaiAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMLanaiDesc.a;/home/siwa/llvm-10.0/lib/libLLVMLanaiInfo.a;/home/siwa/llvm-10.0/lib/libLLVMHexagonDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMHexagonCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMHexagonAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMHexagonDesc.a;/home/siwa/llvm-10.0/lib/libLLVMHexagonInfo.a;/home/siwa/llvm-10.0/lib/libLLVMBPFDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMBPFCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMBPFAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMBPFDesc.a;/home/siwa/llvm-10.0/lib/libLLVMBPFInfo.a;/home/siwa/llvm-10.0/lib/libLLVMARMDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMARMCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMARMAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMARMDesc.a;/home/siwa/llvm-10.0/lib/libLLVMARMUtils.a;/home/siwa/llvm-10.0/lib/libLLVMARMInfo.a;/home/siwa/llvm-10.0/lib/libLLVMAMDGPUDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMAMDGPUCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMMIRParser.a;/home/siwa/llvm-10.0/lib/libLLVMipo.a;/home/siwa/llvm-10.0/lib/libLLVMInstrumentation.a;/home/siwa/llvm-10.0/lib/libLLVMVectorize.a;/home/siwa/llvm-10.0/lib/libLLVMLinker.a;/home/siwa/llvm-10.0/lib/libLLVMIRReader.a;/home/siwa/llvm-10.0/lib/libLLVMAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMAMDGPUAsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMAMDGPUDesc.a;/home/siwa/llvm-10.0/lib/libLLVMAMDGPUUtils.a;/home/siwa/llvm-10.0/lib/libLLVMAMDGPUInfo.a;/home/siwa/llvm-10.0/lib/libLLVMAArch64Disassembler.a;/home/siwa/llvm-10.0/lib/libLLVMMCDisassembler.a;/home/siwa/llvm-10.0/lib/libLLVMAArch64CodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMCFGuard.a;/home/siwa/llvm-10.0/lib/libLLVMGlobalISel.a;/home/siwa/llvm-10.0/lib/libLLVMSelectionDAG.a;/home/siwa/llvm-10.0/lib/libLLVMAsmPrinter.a;/home/siwa/llvm-10.0/lib/libLLVMDebugInfoDWARF.a;/home/siwa/llvm-10.0/lib/libLLVMCodeGen.a;/home/siwa/llvm-10.0/lib/libLLVMTarget.a;/home/siwa/llvm-10.0/lib/libLLVMScalarOpts.a;/home/siwa/llvm-10.0/lib/libLLVMInstCombine.a;/home/siwa/llvm-10.0/lib/libLLVMAggressiveInstCombine.a;/home/siwa/llvm-10.0/lib/libLLVMTransformUtils.a;/home/siwa/llvm-10.0/lib/libLLVMBitWriter.a;/home/siwa/llvm-10.0/lib/libLLVMAnalysis.a;/home/siwa/llvm-10.0/lib/libLLVMProfileData.a;/home/siwa/llvm-10.0/lib/libLLVMObject.a;/home/siwa/llvm-10.0/lib/libLLVMTextAPI.a;/home/siwa/llvm-10.0/lib/libLLVMBitReader.a;/home/siwa/llvm-10.0/lib/libLLVMCore.a;/home/siwa/llvm-10.0/lib/libLLVMRemarks.a;/home/siwa/llvm-10.0/lib/libLLVMBitstreamReader.a;/home/siwa/llvm-10.0/lib/libLLVMAArch64AsmParser.a;/home/siwa/llvm-10.0/lib/libLLVMMCParser.a;/home/siwa/llvm-10.0/lib/libLLVMAArch64Desc.a;/home/siwa/llvm-10.0/lib/libLLVMMC.a;/home/siwa/llvm-10.0/lib/libLLVMDebugInfoCodeView.a;/home/siwa/llvm-10.0/lib/libLLVMDebugInfoMSF.a;/home/siwa/llvm-10.0/lib/libLLVMBinaryFormat.a;/home/siwa/llvm-10.0/lib/libLLVMAArch64Utils.a;/home/siwa/llvm-10.0/lib/libLLVMAArch64Info.a;/home/siwa/llvm-10.0/lib/libLLVMSupport.a;/home/siwa/llvm-10.0/lib/libLLVMDemangle.a;-lrt;-ldl;-ltinfo;-lpthread;-lm
-- Found TVM_LLVM_VERSION=100
-- Build with LLVM 
-- Set TVM_LLVM_VERSION=100
-- Using BLAS library /usr/lib/x86_64-linux-gnu/libopenblas.so
-- Use MKL library /opt/intel/oneapi/mkl/latest/lib/intel64/libmkl_rt.so
-- Use MKLDNN library /usr/local/lib/libdnnl.so
-- Build with DNNL JSON runtime: /usr/local/lib/libdnnl.so
-- Build with contrib.random
-- Build with contrib.sort
-- Build with contrib.hybriddump
-- Git found: /usr/bin/git
-- Found TVM_GIT_COMMIT_HASH=701d2c32759c95951c4eeca229addd1036539698
-- Building with TVM Map...
-- Build with thread support...
-- Configuring done
-- Generating done
-- Build files have been written to: /home/siwa/tvm_official/build

I can provide mine. However, I can run with DNNL runtime now with different model from before, so I guess DNNL have some problem with one of my models.

P.S. I use DNNL 2.2 and TVM 0.7.0 tag

SergeS · August 10, 2021, 9:03am

There is no problems with CMake config but there is not enough information about the model you tried to run. Is this one of the SOTA models?

juierror · August 11, 2021, 2:21am

I see. The model I use is not a SOTA models, but it is some transformer models which use in my company.

jszaday · August 11, 2021, 3:01am

Can you successfully run tests/python/relay/test_external_codegen.py now? If so, what did you change to facilitate it? Reverting to the TVM v0.7.0 tag from upstream main? (The latter is what I was testing on.)

juierror · August 11, 2021, 3:09am

In my case, I just run the model and get the same error as your post. However, in my case, the error has gone after I change models.

P.S. I also try run the test python3 ./test_external_codegen.py on TVM 0.7 tag and didn’t see any error.