[ONNX] error when using nnvm.frontend.from_onnx(onnx_model) on FPGA

1

When I am trying to load the parameters and graph on nnvm this error occurs, I have no problem running them on x86 or when I am using mxnet to import the models.The code is the same as the one provided on the tutorials. I have just added the onnx.load and the from_onnx function call:

 [12:11:32] /data/tools/mxnet/dmlc-core/include/dmlc/./logging.h:308: [12:11:32] /data/tools/mxnet/dmlc-core/include/dmlc/any.h:289: Check failed: type_->ptype_info == &typeid(T) The stored type mismatch stored=NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE requested=NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

    Stack trace returned 7 entries:
    [bt] (0) /usr/local/lib/python3.5/dist-packages/mxnet-0.11.0-py3.5.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x21) [0xb35de1ba]
    [bt] (1) /usr/local/lib/python3.5/dist-packages/mxnet-0.11.0-py3.5.egg/mxnet/libmxnet.so(_ZNK4dmlc3any10check_typeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvv+0x1c3) [0xb3ea735c]
    [bt] (2) /usr/local/lib/python3.5/dist-packages/mxnet-0.11.0-py3.5.egg/mxnet/libmxnet.so(_ZN4dmlc4json14AnyJSONManager8WriteAnyINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvPNS_10JSONWriterERKNS_3anyE+0x1b) [0xb3ebbf3c]
    [bt] (3) /usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/libnnvm_compiler.so(_ZN4dmlc4json7HandlerINS_3anyEE5WriteEPNS_10JSONWriterERKS2_+0x4f1) [0xb1277a1a]
    [bt] (4) /usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/libnnvm_compiler.so(NNGraphGetJSONAttr+0x177) [0xb1272358]
    [bt] (5) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-arm-linux-gnueabihf.so(ffi_call_VFP+0x54) [0xb69f1534]
    [bt] (6) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-arm-linux-gnueabihf.so(ffi_call+0xc9) [0xb69f111e]

    Traceback (most recent call last):
      File "imagenet_classify.py", line 183, in <module>
        main()
      File "imagenet_classify.py", line 84, in main
        net, params = nnvm.frontend.from_onnx(onnx_model)
      File "/usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/frontend/onnx.py", line 810, in from_onnx
        sym, params = g.from_onnx(graph, opset)
      File "/usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/frontend/onnx.py", line 665, in from_onnx
        op = self._convert_operator(op_name, inputs, attr, opset)
      File "/usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/frontend/onnx.py", line 766, in _convert_operator
        sym = convert_map[op_name](inputs, attrs, self._params)
      File "/usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/frontend/onnx.py", line 122, in _impl_v1
        channels = _infer_channels(inputs[1], params)
      File "/usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/frontend/onnx.py", line 393, in _infer_channels
        _, out_shapes = graph_util.infer_shape(g, **shape_dict)
      File "/usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/compiler/graph_util.py", line 30, in infer_shape
        graph = graph_attr.set_shape_inputs(graph, shape)
      File "/usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/compiler/graph_attr.py", line 24, in set_shape_inputs
        shape.get(name, ()) for name in g.index.input_names]
      File "/usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/graph.py", line 194, in index
        self._index = GraphIndex(self)
      File "/usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/graph.py", line 26, in __init__
        jgraph = json.loads(create(graph).apply("SaveJSON").json_attr("json"))
      File "/usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/graph.py", line 122, in json_attr
        self.handle, c_str(key), ctypes.byref(ret), ctypes.byref(success)))
      File "/usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/_base.py", line 75, in check_call
        raise NNVMError(py_str(_LIB.NNGetLastError()))
    nnvm._base.NNVMError: [12:11:32] /data/tools/mxnet/dmlc-core/include/dmlc/any.h:289: Check failed: type_->ptype_info == &typeid(T) The stored type mismatch stored=NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE requested=NSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

    Stack trace returned 7 entries:
    [bt] (0) /usr/local/lib/python3.5/dist-packages/mxnet-0.11.0-py3.5.egg/mxnet/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x21) [0xb35de1ba]
    [bt] (1) /usr/local/lib/python3.5/dist-packages/mxnet-0.11.0-py3.5.egg/mxnet/libmxnet.so(_ZNK4dmlc3any10check_typeINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvv+0x1c3) [0xb3ea735c]
    [bt] (2) /usr/local/lib/python3.5/dist-packages/mxnet-0.11.0-py3.5.egg/mxnet/libmxnet.so(_ZN4dmlc4json14AnyJSONManager8WriteAnyINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEvPNS_10JSONWriterERKNS_3anyE+0x1b) [0xb3ebbf3c]
    [bt] (3) /usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/libnnvm_compiler.so(_ZN4dmlc4json7HandlerINS_3anyEE5WriteEPNS_10JSONWriterERKS2_+0x4f1) [0xb1277a1a]
    [bt] (4) /usr/local/lib/python3.5/dist-packages/nnvm-0.8.0-py3.5.egg/nnvm/libnnvm_compiler.so(NNGraphGetJSONAttr+0x177) [0xb1272358]
    [bt] (5) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-arm-linux-gnueabihf.so(ffi_call_VFP+0x54) [0xb69f1534]
    [bt] (6) /usr/lib/python3.5/lib-dynload/_ctypes.cpython-35m-arm-linux-gnueabihf.so(ffi_call+0xc9) [0xb69f111e]
2

Hi @cntontis, thank you for reporting the ONNX import bug. Right now graph support for ONNX (and other formats) for the FPGA is work in progress but not supported as of now. @tqchen - do you care to comment about when the patch will be available?

3

@thierry maybe the FPGA term that I used is misleading. The board has an arm Cpu and I am running ubuntu on it. Is it still the same?

4

@cntontis which board specifically are you using? Is it the Pynq board? or another board? Thank you.

5

Its the xilinx zc706

6

I see, we’re currently not supporting this board as of now for the FPGA back end but adding another FPGA device should not require too much work given that these are similar SoCs.

If you want to obtain a CPU baseline, I recommend that you do not build upon the VTA tutorial (https://docs.tvm.ai/vta/tutorials/resnet.html#sphx-glr-vta-tutorials-resnet-py) for now since it is built for Pynq and has narrow support for other model entry methods (but we’re actively working on it so that other models can be easily ported).

For more information on importing ONNX with TVM: https://docs.tvm.ai/tutorials/nnvm/from_onnx.html#sphx-glr-tutorials-nnvm-from-onnx-py

Hope this helps

7

This exactly the tutorial that I am using (the Last one that you mentioned ) I Just changed the target to “llvm”. So if I understoud well, The problem is That the backend of nnvm doesn’t support the onnx format for the specific architecture of arm cpu?

8

Due to restrictions of FPGA accelerators, that operates on fixed point integer, we yet cannot directly take in arbitrary floating point models. Hopefully we will have updates and improve the state so that things will be better

9

I see, so I can use only the mxnet models for now. Thank you very much for your responses and your patience :grin: