[SOLVED] OpenCL problem with Hikey970

tkclimb · September 14, 2018, 5:06am

I tried to execute simple workload on Hikey970 with OpenCL (Mali-G72 MP12) throughout RPC, just referencing https://docs.tvm.ai/tutorials/cross_compilation_and_rpc.html#sphx-glr-tutorials-cross-compilation-and-rpc-py.

In the experiments, local CPU (macOS) and remote CPU (aarch64) worked, however the remote CPU with OpenCL hanged up with the following error message.

# stdout on remote (Hikey970)
[20:28:24] /home/shunya/repos/tvm/src/runtime/opencl/opencl_device_api.cc:255: Initialize OpenCL platform 'ARM Platform'
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded

# stdout on host (mac)
 yamada  …  workspace  tvm  resnet  python tvm_deploy_multiple_targets.py -t hikey970 -cl
[23:58:24] /Users/yamada/tk-home/workspace/tvm/tvm/src/codegen/opt/build_opencl_off.cc:16: OpenCL runtime not enabled, return a source module...
Traceback (most recent call last):
  File "tvm_deploy_multiple_targets.py", line 129, in <module>
    run_opencl()
  File "tvm_deploy_multiple_targets.py", line 88, in run_opencl
    func = remote.load_module(lib_name)
  File "/Users/yamada/tk-home/workspace/tvm/tvm/python/tvm/rpc/client.py", line 119, in load_module
    return base._LoadRemoteModule(self._sess, path)
  File "/Users/yamada/tk-home/workspace/tvm/tvm/python/tvm/_ffi/function.py", line 280, in my_api_func
    return flocal(*args)
  File "/Users/yamada/tk-home/workspace/tvm/tvm/python/tvm/_ffi/_ctypes/function.py", line 184, in __call__
    ctypes.byref(ret_val), ctypes.byref(ret_tcode)))
  File "/Users/yamada/tk-home/workspace/tvm/tvm/python/tvm/_ffi/base.py", line 66, in check_call
    raise TVMError(py_str(_LIB.TVMGetLastError()))
tvm._ffi.base.TVMError: Except caught from RPC call: TVMCall CFunc Error:
Traceback (most recent call last):
  File "/home/shunya/repos/tvm/python/tvm/_ffi/_ctypes/function.py", line 55, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/shunya/repos/tvm/python/tvm/rpc/server.py", line 50, in load_module
    m = _load_module(path)
  File "/home/shunya/repos/tvm/python/tvm/module.py", line 225, in load
    return _LoadFromFile(path, fmt)
  File "/home/shunya/repos/tvm/python/tvm/_ffi/_ctypes/function.py", line 185, in __call__
    ctypes.byref(ret_val), ctypes.byref(ret_tcode)))
  File "/home/shunya/repos/tvm/python/tvm/_ffi/base.py", line 66, in check_call
    raise TVMError(py_str(_LIB.TVMGetLastError()))
tvm._ffi.base.TVMError: [20:28:24] /home/shunya/repos/tvm/src/runtime/opencl/opencl_device_api.cc:269: Check failed: err_code == CL_SUCCESS OpenCL Error, code=-6: CL_OUT_OF_HOST_MEMORY

Stack trace returned 10 entries:
[bt] (0) /home/shunya/repos/tvm/build/libtvm_runtime.so(dmlc::StackTrace[abi:cxx11]()+0x104) [0xffff7b95e1c4]
[bt] (1) /home/shunya/repos/tvm/build/libtvm_runtime.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x30) [0xffff7b95eca8]
[bt] (2) /home/shunya/repos/tvm/build/libtvm_runtime.so(tvm::runtime::cl::OpenCLWorkspace::Init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x224) [0xffff7b9c91fc]
[bt] (3) /home/shunya/repos/tvm/build/libtvm_runtime.so(tvm::runtime::OpenCLModuleNode::Init()+0x134) [0xffff7b9d482c]
[bt] (4) /home/shunya/repos/tvm/build/libtvm_runtime.so(tvm::runtime::OpenCLModuleCreate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, tvm::runtime::FunctionInfo, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, tvm::runtime::FunctionInfo> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x27c) [0xffff7b9d4f6c]
[bt] (5) /home/shunya/repos/tvm/build/libtvm_runtime.so(tvm::runtime::OpenCLModuleLoadBinary(void*)+0x208) [0xffff7b9d6000]
[bt] (6) /home/shunya/repos/tvm/build/libtvm_runtime.so(+0x9a3fc) [0xffff7b9d63fc]
[bt] (7) /home/shunya/repos/tvm/build/libtvm_runtime.so(tvm::runtime::ImportModuleBlob(char const*, std::vector<tvm::runtime::Module, std::allocator<tvm::runtime::Module> >*)+0x780) [0xffff7b983f58]
[bt] (8) /home/shunya/repos/tvm/build/libtvm_runtime.so(+0x30194) [0xffff7b96c194]
[bt] (9) /home/shunya/repos/tvm/build/libtvm_runtime.so(tvm::runtime::Module::LoadFromFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x3b0) [0xffff7b97c628]

If my understanding is correct, OpenCL 2.1 is installed in the linux image I’m using on Hikey970.
device file /dev/mali0 exists.

tkclimb · September 2, 2018, 3:13pm

OK, I found the kernel module may not have been loaded in this image.
I referenced clinfo, I paste it also blow just in case.

shunya@hikey970:~/repos/tvm$ clinfo
Number of platforms                               1
  Platform Name                                   ARM Platform
  Platform Vendor                                 ARM
  Platform Version                                OpenCL 2.0 v1.r10p0-01rel0.e990c3e3ae25bde6c6a1b96097209d52
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_3d_image_writes cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_fp16 cl_khr_icd cl_khr_egl_image cl_khr_image2d_from_buffer cl_khr_depth_images cl_arm_core_id cl_arm_printf cl_arm_thread_limit_hint cl_arm_non_uniform_work_group_size cl_arm_import_memory cl_arm_shared_virtual_memory
  Platform Extensions function suffix             ARM

  Platform Name                                   ARM Platform
Number of devices                                 1
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Device Name                                     <printDeviceInfo:0: get Device Name size : error -6>
  Device Vendor                                   ARM
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Device Vendor ID                                <printDeviceInfo:2: get CL_DEVICE_VENDOR_ID : error -6>
  Device Version                                  OpenCL 2.0 v1.r10p0-01rel0.e990c3e3ae25bde6c6a1b96097209d52
  Driver Version                                  2.0
  Device OpenCL C Version                         OpenCL C 2.0 v1.r10p0-01rel0.e990c3e3ae25bde6c6a1b96097209d52
  Device Type                                     GPU
  Device Profile                                  FULL_PROFILE
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Max compute units                               <printDeviceInfo:14: get CL_DEVICE_MAX_COMPUTE_UNITS : error -6>
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Max clock frequency                             <printDeviceInfo:18: get CL_DEVICE_MAX_CLOCK_FREQUENCY : error -6>
  Device Partition                                (core)
    Max number of sub-devices                     0
    Supported partition types                     None
  Max work item dimensions                        3
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Max work item sizes                             <printDeviceInfo:33: get number of CL_DEVICE_MAX_WORK_ITEM_SIZES : error -6>
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Max work group size                             <printDeviceInfo:34: get CL_DEVICE_MAX_WORK_GROUP_SIZE : error -6>
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Preferred work group size multiple              <getWGsizes:494: create context : error -6>
  Preferred / native vector sizes
    char                                                16 / 4
    short                                                8 / 2
    int                                                  4 / 1
    long                                                 2 / 1
    half                                                 8 / 2        (cl_khr_fp16)
    float                                                4 / 1
    double                                               0 / 0        (n/a)
  Half-precision Floating-point support           (cl_khr_fp16)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  No
  Single-precision Floating-point support         (core)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  No
  Double-precision Floating-point support         (n/a)
  Address bits                                    64, Little-Endian
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Global memory size                              <printDeviceInfo:67: get CL_DEVICE_GLOBAL_MEM_SIZE : error -6>
  Error Correction support                        No
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Max memory allocation                           <printDeviceInfo:73: get CL_DEVICE_MAX_MEM_ALLOC_SIZE : error -6>
  Unified memory for Host and Device              Yes
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Shared Virtual Memory (SVM) capabilities        <printDeviceInfo:76: get CL_DEVICE_SVM_CAPABILITIES : error -6>
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       1024 bits (128 bytes)
  Preferred alignment for atomics
    SVM                                           0 bytes
    Global                                        0 bytes
    Local                                         0 bytes
  Max size for global variable                    65536 (64KiB)
  Preferred total size of global vars             0
  Global Memory cache type                        Read/Write
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Global Memory cache size                        <printDeviceInfo:89: get CL_DEVICE_GLOBAL_MEM_CACHE_SIZE : error -6>
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Global Memory cache line                        <printDeviceInfo:90: get CL_DEVICE_GLOBAL_MEM_CACHELINE_SIZE : error -6>
  Image support                                   Yes
    Max number of samplers per kernel             16
    Max size for 1D images from buffer            65536 pixels
    Max 1D or 2D image array size                 2048 images
    Base address alignment for 2D image buffers   32 bytes
    Pitch alignment for 2D image buffers          64 bytes
    Max 2D image size                             65536x65536 pixels
    Max 3D image size                             65536x65536x65536 pixels
    Max number of read image args                 128
    Max number of write image args                64
    Max number of read/write image args           64
  Max number of pipe args                         16
  Max active pipe reservations                    1
  Max pipe packet size                            1024
  Local memory type                               Global
  Local memory size                               32768 (32KiB)
  Max constant buffer size                        65536 (64KiB)
  Max number of constant args                     8
  Max size of kernel argument                     1024
  Queue properties (on host)
    Out-of-order execution                        Yes
    Profiling                                     Yes
  Queue properties (on device)
    Out-of-order execution                        Yes
    Profiling                                     Yes
    Preferred size                                2097152 (2MiB)
    Max size                                      16777216 (16MiB)
  Max queues on device                            1
  Max events on device                            1024
  Prefer user sync for interop                    No
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  Profiling timer resolution                      <printDeviceInfo:128: get CL_DEVICE_PROFILING_TIMER_RESOLUTION : error -6>
  Execution capabilities
    Run OpenCL kernels                            Yes
    Run native kernels                            No
  printf() buffer size                            1048576 (1024KiB)
  Built-in kernels
  Device Available                                Yes
  Compiler Available                              Yes
  Linker Available                                Yes
  Device Extensions                               cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_khr_3d_image_writes cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_fp16 cl_khr_icd cl_khr_egl_image cl_khr_image2d_from_buffer cl_khr_depth_images cl_arm_core_id cl_arm_printf cl_arm_thread_limit_hint cl_arm_non_uniform_work_group_size cl_arm_import_memory cl_arm_shared_virtual_memory

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  ARM Platform
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   Success [ARM]
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  clCreateContext(NULL, ...) [default]            <checkNullCtx:2241: create context with device from default platform : error -6>
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  <checkNullCtxFromType:2284: create context from type CL_DEVICE_TYPE_GPU : error -6>
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
Failed creating base context during opening of kernel driver.
Kernel module may not have been loaded
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  <checkNullCtxFromType:2284: create context from type CL_DEVICE_TYPE_ALL : error -6>

ICD loader properties
  ICD loader Name                                 OpenCL ICD Loader
  ICD loader Vendor                               OCL Icd free software
  ICD loader Version                              2.2.11
  ICD loader Profile                              OpenCL 2.1

Is there any one ran some workload on Hikey970 with OpenCL in TVM community??

tkclimb · September 14, 2018, 5:05am

I made a simple mistake that I needed to run the server process with root privilege.
so, the command on server side should be sudo python -m tvm.exec.rpc_server --host 0.0.0.0 --port=9090.