Can I get a inference time layer by layer?

Questions
#1

Hello, I’m currently working on a simulating inferencing resnet18 using relay.build and graphexecuter. I got the whole inference time using “graph_executor.GraphModule.benchmark”, but I wonder there’s a way to get a infernece time layer by layer.

here is the main part of the code target = “llvm -mcpu=skylake-avx512” input_name = “data” shape_dict = {input_name: img_data.shape}

save_path = Path("/home/sungrae/khb/TVM/compiled_quantized_model/")
path_lib = save_path / "deploy_original_lib.tar"

lib = tvm.runtime.load_module(path_lib)

dev = tvm.device(str(target), 0)
module = graph_executor.GraphModule(lib["default"](dev))

dtype = "float32"
module.set_input(input_name, img_data)


log = module.benchmark(dev, func_name='run', repeat=10, number=10, min_repeat_ms=None, limit_zero_time_iterations=100, end_to_end=False, cooldown_interval_ms=0, repeats_to_cooldown=1)
print(log)

Thanks

#2

Perhaps you could consider replacing graph_executor with tvm.contrib.debugger.debug_executor?

1 Like
#3

i think the most accurate tool is performance tool like Intel VTune for Intel CPU, Nvidia Nsight Systems for Nvidia GPU. debug_executor could give you a brief report, but it is not the most accurate especially you are on a heterogeneous hardware like GPU, CPU is a little bit better.

2 Likes
#4

I’m So Soo thankful for you reply. As you suggested I tried uising

tvm.contrib.debugger.debug_executor.GraphModuleDebug

and it says it needs graph_json_str. I tried so hard finding how to get this str info but I couldn’t. I guess this is some kind of basic format file and tvm.relay.build’s output might includes it??

code is changed like below

lib = tvm.runtime.load_module(path_lib)
graph_json = lib.GraphExecutorFactory
print(graph_json)
# graph_json = lib.graph_json
# print(graph_json)
#graph_json = tvm.relay.build(lib)
dev = tvm.device(str(target), 0)
tvm.contrib.graph_executor.create
#module = graph_executor.GraphModule(lib["default"](dev))
module = tvm.contrib.debugger.debug_executor.GraphModuleDebug(lib["default"](dev), dev, graph_json)
#5

Turns out relay.build.get_graph_json was the answer. here’s the new code

batch_size = 1
num_class = 1000
image_shape = (3, 224, 224)
data_shape = (batch_size,) + image_shape
out_shape = (batch_size, num_class)

mod, params = relay.testing.resnet.get_workload(
    num_layers=18, batch_size=batch_size, image_shape=image_shape
)
target = "llvm -mcpu=skylake-avx512"
lib = relay.build(mod, target=target, params=params)
get_graph_json = lib.get_graph_json
print(get_graph_json)
dev = tvm.device(str(target), 0)
tvm.contrib.graph_executor.create
module = graph_executor.GraphModule(lib["default"](dev))
module = tvm.contrib.debugger.debug_executor.GraphModuleDebug(lib["default"](dev), dev, get_graph_json, '/home/sungrae/khb/TVM/dumproot')

dtype = "float32"
module.set_input(input_name, img_data)

log = module.benchmark(dev, func_name='run', repeat=10, number=10, min_repeat_ms=None, limit_zero_time_iterations=100, end_to_end=False, cooldown_interval_ms=0, repeats_to_cooldown=1)
print(log)

But then this comes out

[17:02:03] /workspace/tvm/src/runtime/threading_backend.cc:343: Warning: more than two frequencies detected! Forced big_count_ to 8
One or more operators have not been tuned. Please tune your model for better performance. Use DEBUG logging level to see more details.
<bound method GraphExecutorFactoryModule.get_graph_json of <tvm.relay.backend.executor_factory.GraphExecutorFactoryModule object at 0x7f86b325ef10>>
Traceback (most recent call last):
  File "/home/sungrae/anaconda3/envs/TVM_Kang/lib/python3.7/site-packages/tvm/relay/Quick Start Tutorial for Compiling Deep Learning Models.py", line 76, in <module>
    module = tvm.contrib.debugger.debug_executor.GraphModuleDebug(lib["default"](dev), dev, get_graph_json, '/home/sungrae/khb/TVM/dumproot')
  File "/home/sungrae/anaconda3/envs/TVM_Kang/lib/python3.7/site-packages/tvm/contrib/debugger/debug_executor.py", line 116, in __init__
    self._run_individual = module["run_individual"]
  File "/home/sungrae/anaconda3/envs/TVM_Kang/lib/python3.7/site-packages/tvm/runtime/module.py", line 193, in __getitem__
    return self.get_function(name)
  File "/home/sungrae/anaconda3/envs/TVM_Kang/lib/python3.7/site-packages/tvm/runtime/module.py", line 177, in get_function
    raise AttributeError(f"Module has no function '{name}'")
AttributeError: Module has no function 'run_individual'

I presume that in order to use GraphModuleDebug, I have to use relay.build differently???

#6

Hello! Here are examples how to use debagger. I suppose the example in paragraph 3 is suitable for your case.

2 Likes
#8

Hello! I really appreciate your reply, and it helped a lot!