Something like this tutorial either using tvm::runtime::Module::GetFunction or TVMFuncListGlobalNames should help?
Yes, but we should know functions present inside module before GetFunction, in tvm tutorial they used an example as mod.GetFunction(“set_input”); here set_input is function name present inside module. My question is how to know the list of such function inside a module ?
@srkreddy1238 @FrozenGene @tqchen Any suggestion on this question?
Dump the TVM graph as JSON which contain func_name attribute which is nothing but the symbol of compiled library. See below ref
{
"nodes": [
{
"op": "null",
"name": "input",
"inputs": []
},
{
"op": "tvm_op",
"name": "fused_nn_pad_19",
"attrs": {
"func_name": "fused_nn_pad_19",
"flatten_data": "0",
"num_inputs": "1",
"num_outputs": "1"
},
Thank you @srkreddy1238
Here function name is “func_name”: “fused_nn_pad_19” ?
and in mod.GetFunction(“"fused_nn_pad_19”); ?
Those are the symbol names using which TVM runtime extract when loaded.
I couldn’t understand your objective of calling them directly. Can you explain ?
@srkreddy1238 Basically i need to Deploy TVM Module using C++ API, for this i should know the function inside the module. My question is how to extract function from module or how to know functions from that module.
Like “set_input”, “set_out” and “run” which i can use for mod.GetFunction(), and do operations.
Just i need function names from tvm module to call those function from C++ API.
I couldn’t imagine the case you should call these functions directly. Basically, when you deploy using C++ API, you could just call set_input , run and get_output. When we call run, TVM will parse deploy_graph.json, find tvm_op and its related compiled function like fuse_conv2d, fuse_conv2d_1. (Every model has unique compiled function). Everything is handled by TVM runtime including data manipulation.
@FrozenGene Thanks for your replay.
After calling run how to call tvm_oprations in C++? like if I want to call fuse_conv2d operation, how to call this?
Is it any customised deployment ?
ref. apps/howto_deploy/
Cpp deployment doesn’t need any details about function names inside the lib.
The run method of graph runtime calls all these functions in a loop.
@srkreddy1238 @FrozenGene When i try to get out from module i am getting below error. I am running tvm tutorial c++ example to deploy module from that example get_output(0, y); is giving below error.
terminate called after throwing an instance of ‘dmlc::Error’
what(): [13:42:14] /home/ubuntu/tvm_opencl/tvm/src/runtime/graph/graph_runtime.cc:121: Check failed: data->shape[j] == data_out->shape[j] (256 vs. 1000)
Stack trace returned 7 entries:
[bt] (0) /usr/lib32/libtvm_runtime.so(+0x1198d) [0x7ffff7b1198d]
[bt] (1) /usr/lib32/libtvm_runtime.so(+0x125dd) [0x7ffff7b125dd]
[bt] (2) /usr/lib32/libtvm_runtime.so(+0x7b214) [0x7ffff7b7b214]
[bt] (3) /usr/lib32/libtvm_runtime.so(+0x7bfc2) [0x7ffff7b7bfc2]
[bt] (4) /home/ubuntu/Mallappa/NNVM/NNVMDeploy/testdeploy() [0x402b04]
[bt] (5) /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7ffff6d9d830]
[bt] (6) /home/ubuntu/Mallappa/NNVM/NNVMDeploy/testdeploy() [0x402fa9]
Thread 1 “testdeploy” received signal SIGABRT, Aborted.
0x00007ffff6db2428 in __GI_raise (sig=sig@entry=6) at …/sysdeps/unix/sysv/linux/raise.c:54
54 …/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
What is the model output type and shape ?
Alternatively don’t pass second argument in which case get_output returns NDArray.
Below is the output model and Shape.
DLTensor* y;
int out_ndim = 2;
int dtype_code = kDLFloat;
int dtype_bits = 32;
int dtype_lanes = 1;
int device_type = 4 ;
int device_id = 0;
int64_t out_shape[2] = {1, 1000, };
TVMArrayAlloc(out_shape, out_ndim, dtype_code, dtype_bits, dtype_lanes, device_type, device_id, &y);
tvm::runtime::PackedFunc get_output = mod.GetFunction("get_output");
CHECK(get_output != nullptr);
get_output(0, y);The output shape seem to be {1, 256}, hence this error.
Thank you @srkreddy1238.
And @srkreddy1238 @FrozenGene i am trying set input as ppm image file to tvm module so i am getting different result for python and c++ i have deployed same tvm module in both python and c++ i am getting different result.
Can you share the graph.json if possible. I will have a look at the input and output signatures.
@srkreddy1238 I can’t share complete json file it’s company confidential i have shared some snap of it.
{
“nodes”: [
{
“op”: “null”,
“name”: “input”,
“inputs”: []
},
{
“op”: “tvm_op”,
“name”: “transpose0”,
“attrs”: {
“flatten_data”: “0”,
“func_name”: “fuse_transpose”,
“num_inputs”: “1”,
“num_outputs”: “1”
},
“inputs”: [[0, 0, 0]]
},
{
“op”: “tvm_op”,
“name”: “pad0”,
“attrs”: {
“flatten_data”: “0”,
“func_name”: “fuse_pad”,
“num_inputs”: “1”,
“num_outputs”: “1”
},
“inputs”: [[1, 0, 0]]
},
{
“op”: “null”,
“name”: “SENet/first_layer/first_layer_conv1/conv2d/kernel”,
“inputs”: []
},
{
“op”: “tvm_op”,
“name”: “transpose1”,
“attrs”: {
“flatten_data”: “0”,
“func_name”: “fuse_transpose_1”,
“num_inputs”: “1”,
“num_outputs”: “1”
},
“inputs”: [[3, 0, 0]]
},
{
“op”: “tvm_op”,
“name”: “SENet_2/first_layer/first_layer_conv1/conv2d/Conv2D”,
“attrs”: {
“flatten_data”: “0”,
“func_name”: “fuse_conv2d”,
“num_inputs”: “2”,
“num_outputs”: “1”
},
“inputs”: [[2, 0, 0], [4, 0, 0]]
},
{
“op”: “tvm_op”,
“name”: “transpose2”,
“attrs”: {
“flatten_data”: “0”,
“func_name”: “fuse_transpose_2”,
“num_inputs”: “1”,
“num_outputs”: “1”
},
“inputs”: [[5, 0, 0]]
},
{
“op”: “null”,
“name”: “SENet/first_layer/first_layer_batch1/moving_variance”,
“inputs”: []
},
{
“op”: “tvm_op”,
“name”: “SENet_2/first_layer/first_layer_batch1/FusedBatchNorm_add_eps”,
“attrs”: {
“flatten_data”: “1”,
“func_name”: “fuse___add_scalar__”,
“num_inputs”: “1”,
“num_outputs”: “1”
},
“inputs”: [[7, 0, 1]]
},
{
“op”: “tvm_op”,
“name”: “SENet_2/first_layer/first_layer_batch1/FusedBatchNorm_sqrt”,
“attrs”: {
“flatten_data”: “1”,
“func_name”: “fuse_sqrt”,
“num_inputs”: “1”,
“num_outputs”: “1”
},
“inputs”: [[8, 0, 0]]
},
{
“op”: “tvm_op”,
“name”: “SENet_2/first_layer/first_layer_batch1/FusedBatchNorm_div”,
“attrs”: {
“flatten_data”: “1”,
“func_name”: “fuse___rdiv_scalar__”,
“num_inputs”: “1”,
“num_outputs”: “1”
},
“inputs”: [[9, 0, 0]]
},
{
“op”: “null”,
“name”: “SENet/first_layer/first_layer_batch1/gamma”,
“inputs”: []
},
{
“op”: “tvm_op”,
“name”: “SENet_2/first_layer/first_layer_batch1/FusedBatchNorm_gamma_mul_div”,
“attrs”: {
“flatten_data”: “1”,
“func_name”: “fuse_elemwise_mul”,
“num_inputs”: “2”,
“num_outputs”: “1”
},
“inputs”: [[10, 0, 0], [11, 0, 0]]
},
{
“op”: “tvm_op”,
“name”: “SENet_2/first_layer/first_layer_batch1/FusedBatchNorm_a_mul_data”,
“attrs”: {
“flatten_data”: “0”,
“func_name”: “fuse_broadcast_mul”,
“num_inputs”: “2”,
“num_outputs”: “1”
},
“inputs”: [[6, 0, 0], [12, 0, 0]]
},
{
“op”: “null”,
“name”: “SENet/first_layer/first_layer_batch1/moving_mean”,
“inputs”: []
},
{
“op”: “tvm_op”,
“name”: “SENet_2/first_layer/first_layer_batch1/FusedBatchNorm_neg_mean”,
“attrs”: {
“flatten_data”: “1”,
“func_name”: “fuse_negative”,
“num_inputs”: “1”,
“num_outputs”: “1”
},
“inputs”: [[14, 0, 1]]
},
{
“op”: “tvm_op”,
“name”: “SENet_2/first_layer/first_layer_batch1/FusedBatchNorm_neg_mean_mul_a”,
“attrs”: {
“flatten_data”: “1”,
“func_name”: “fuse_elemwise_mul”,
“num_inputs”: “2”,
“num_outputs”: “1”
},
“inputs”: [[15, 0, 0], [12, 0, 0]]
},
{
“op”: “null”,
“name”: “SENet/first_layer/first_layer_batch1/beta”,
“inputs”: []
},
{
“op”: “tvm_op”,
“name”: “SENet_2/first_layer/first_layer_batch1/FusedBatchNorm_add_beta”,
“attrs”: {
“flatten_data”: “1”,
“func_name”: “fuse_elemwise_add”,
“num_inputs”: “2”,
“num_outputs”: “1”
},
“inputs”: [[16, 0, 0], [17, 0, 0]]
},
{
“op”: “tvm_op”,
“name”: “SENet_2/first_layer/first_layer_batch1/FusedBatchNorm_out”,
“attrs”: {
“flatten_data”: “0”,
“func_name”: “fuse_broadcast_add”,
“num_inputs”: “2”,
“num_outputs”: “1”
},
“inputs”: [[13, 0, 0], [18, 0, 0]]
},
{
“op”: “tvm_op”,
“name”: “SENet_2/first_layer/Relu”,
“attrs”: {
“flatten_data”: “1”,
“func_name”: “fuse_relu”,
“num_inputs”: “1”,
“num_outputs”: “1”
},
“inputs”: [[19, 0, 0]]
},
{
“op”: “tvm_op”,
“name”: “transpose3”,
“attrs”: {
“flatten_data”: “0”,
“func_name”: “fuse_transpose_3”,
“num_inputs”: “1”,
“num_outputs”: “1”
},
“inputs”: [[20, 0, 0]]
},
{
“op”: “tvm_op”,
“name”: “pad1”,
“attrs”: {
“flatten_data”: “0”,
“func_name”: “fuse_pad_1”,
“num_inputs”: “1”,
“num_outputs”: “1”
},
“inputs”: [[21, 0, 0]]
},
I can understand it’s confidential and i don’t want to know the complete graph.
Just look for shape and dltype section for input and output nodes.
For example InceptionV3 has got info like below.
"dltype": ["list_str", [
"float32",
"float32",
"float32",
- - - - - - - --
"float32",
"float32",
"float32",
"float32"]],
"shape": ["list_shape", [
[1, 299, 299, 3],
[3, 3, 3, 32],
-------
[1, 1001],
[1, 1001],
[1, 1001]]]
}
}
Here
input shape is [1, 299, 299, 3] with dtype ‘float32’
output shape is [1, 1001] with dtype ‘float32’
check these sections and allocate the out array accordingly.
Thank you @srkreddy1238. i understood graph.
Do you know how to convert 3d array to 4d array using Mat::reshape in opencv like in python,
x = Image.open(‘cat,png’)
x = np.array(x)
x = np.reshape(x, (1,64,64,3))
basically here ‘x’ is 3D RGB array then they are reshaping it 4D array using numpy.reshape in python, similarly can i do in C++ using Mat::reshape() in opencv?
Thank you.
If your requirement is to copy to NDArray you may use flattened to 1D array and TVMArrayCopyFromBytes API directly. Hence you may not need expand dimensions / reshape.
In your case for C++ all you need to do is read cat.png decode it into a buffer of size 64x64x3 bytes and use TVMArrayCopyFromBytes to set the input into NDArray.
how to dump this graph?