Run Relay-Generated Binary

Questions
#1

Hi, I am new to TVM. I need to convert ONNX models into Relay models, compile Relay models into binary files and then run the binary file in tvm.

My question lies in running the binary file in tvm. I’ve successfully run binary files generated from Relax models through tvm.runtime.relax_vm.VirtualMachine. I’ve tried to operate binary files generated from Relay models through tvm.contrib.graph_executor but failed because I can’t find the main function.

Here’s my code:

import numpy as np
import tvm
from tvm.contrib import graph_executor

lib = tvm.runtime.load_module('output.so')
dev = tvm.cpu(0)
target = 'llvm'

module = graph_executor.create(lib['main'](dev))

X = np.array([[1, 2], [3, 4]], dtype=np.float64)
A = np.array([[5, 6], [7, 8]], dtype=np.float64)
B = np.array([[9, 10], [11, 12]], dtype=np.float64)
C = np.array([[13, 14], [15, 16]], dtype=np.float64)
D = np.array([[17, 18], [19, 20]], dtype=np.float64)

module.set_input("X", X)
module.set_input("A", A)
module.set_input("B", B)
module.set_input("C", C)
module.set_input("D", D)

module.run()
output = module.get_output(0).asnumpy()
print("Output Y:", output)

Following is the error message:

Traceback (most recent call last):
  File "/home/code/Code_TVM/onnx_to_llvm_ir/relay/run_binary.py", line 9, in <module>
    module = graph_executor.create(lib['main'](dev))
                                   ~~~^^^^^^^^
  File "/home/code/tvm/python/tvm/runtime/module.py", line 192, in __getitem__
    return self.get_function(name)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/code/tvm/python/tvm/runtime/module.py", line 176, in get_function
    raise AttributeError(f"Module has no function '{name}'")
AttributeError: Module has no function 'main'

And here’s the Relay model:

def @main(%X: Tensor[(2, 2), float64] /* ty=Tensor[(2, 2), float64] span=MatMul_0.X:0:0 */, %A: Tensor[(2, 2), float64] /* ty=Tensor[(2, 2), float64] span=MatMul_0.A:0:0 */, %B: Tensor[(2, 2), float64] /* ty=Tensor[(2, 2), float64] span=Add_0.B:0:0 */, %C: Tensor[(2, 2), float64] /* ty=Tensor[(2, 2), float64] span=MatMul_1.C:0:0 */, %D: Tensor[(2, 2), float64] /* ty=Tensor[(2, 2), float64] span=Add_1.D:0:0 */) -> Tensor[(2, 2), float64] {
  %0 = transpose(%A, axes=[1, 0]) /* ty=Tensor[(2, 2), float64] span=MatMul_0:0:0 */;
  %1 = nn.dense(%X, %0, units=None, out_dtype="float64") /* ty=Tensor[(2, 2), float64] span=MatMul_0:0:0 */;
  %2 = add(%1, %B) /* ty=Tensor[(2, 2), float64] span=Add_0:0:0 */;
  %3 = nn.relu(%2) /* ty=Tensor[(2, 2), float64] span=Relu_0:0:0 */;
  %4 = transpose(%C, axes=[1, 0]) /* ty=Tensor[(2, 2), float64] span=MatMul_1:0:0 */;
  %5 = nn.dense(%3, %4, units=None, out_dtype="float64") /* ty=Tensor[(2, 2), float64] span=MatMul_1:0:0 */;
  add(%5, %D) /* ty=Tensor[(2, 2), float64] span=Add_1:0:0 */
}

I wonder which function should I get to run the entire model? Are there any possible ways to run the binary file?

Thank you very much!

#2

We are phasing out Relay, please try moving towards to relax

#3

Thank you for your information. But I still wonder what may have caused this problem? And I’d like to know if it’s possible to export the LLVM IR code of a Relax model

#4

You can try this way.

module = graph_executor.GraphModule(lib["default"](dev))
#5

Thank you very much! It works!