Tvm/tutorials/nnvm/form_tensorflow.py Inference time is too long

Questions
1

@srkreddy1238 when i used tvm/tutorials/nnvm/from_tensorflow.py . I find that the tvm inference time is too long.

tvm inference time: 15.335524s

and tf inference time : 3.372391s

Does I have something wrong? Does anyone can give me some advise? Thanks in advance.

diff:
+import time
from tvm.contrib import graph_runtime
dtype = ‘uint8’
+
+start = time.clock()
+
m = graph_runtime.create(graph, lib, ctx)
m.set_input(‘DecodeJpeg/contents’, tvm.nd.array(x.astype(dtype)))
m.set_input(**params)
m.run()
+
+end = time.clock()
+print “execute time:”
+print (str(end - start))
tvm_output = m.get_output(0, tvm.nd.empty(((1, 1008)), ‘float32’))

@@ -194,6 +217,7 @@ def run_inference_on_image(image):
+start = time.clock()
if not tf.gfile.Exists(image):
tf.logging.fatal(‘File does not exist %s’, image)
image_data = tf.gfile.FastGFile(image, ‘rb’).read()
@@ -219,5 +243,8 @@ def run_inference_on_image(image):
human_string = node_lookup.id_to_string(node_id)
score = predictions[node_id]
print(’%s (score = %.5f)’ % (human_string, score))
+end = time.clock()
+print “tf execute time:”
+print (str(end - start))

2

I think we shouldn’t consider m = graph_runtime.create(graph, lib, ctx)
instead try saving compiled module and reload as shown below.

temp = util.tempdir()
path_lib = temp.relpath("deploy.so")
lib.export_library(path_lib)
with open(temp.relpath("deploy.json"), "w") as fo:
    fo.write(graph.json())
with open(temp.relpath("deploy.params"), "wb") as fo:
    fo.write(nnvm.compiler.save_param_dict(params))
print(temp.listdir())

start = time.clock()
loaded_lib = tvm.module.load(path_lib)
loaded_json = open(temp.relpath("deploy.json")).read()
loaded_params = bytearray(open(temp.relpath("deploy.params"), "rb").read())

Also we should tune using AutoTVM to have comparable results with tensorflow.

3

OK, Thanks,

i will tune the tf model using autotvm.