@tqchen thanks.In our case it is distilled bert, the sequence length and hidden size are small, so the computation workload is not that large, and we want to take advantages of tvm’s operator fusion’s capabilities to speedup execution.
And the batch size is dynamic.
By our profiling results, the overhead introduced by relay vm in dynamic mode (5.52ms, BatchSize=1,Relay Batch Dim=Any) is unacceptable compared to static mode(1.24ms, BatchSize=1, Relay Batch Dim=1)
We have made a detailed analysis, the overhead comes mainly from three parts:
- Relay Ops introduced by relay frontend, in BatchMatmul and StridedSlice, we add some new ops to do something like dynamic reshape, this ops are placed on GPU by default. TVM Ops(fused) increased to 2.24X to static mode.
- TVM Ops introduced by relay pass ManifestAlloc to calculate memory assignment size at relay level, including shape functions and storage size calculation ops, this ops are placed on CPU by default. TVM Ops(fused) increased to 5.92X to static mode.
- Instructions introduced by Relay VM, mainly load_const, alloc_tensor and alloc_storage, Instructions increased to 8.90X to static mode, the overhead introduced by executing vm’s instruction is also unacceptable.
Because of 1&2, after all relay passes is done and compiled to VM Bytecodes, there are left large amount of small pieces of tvm ops(PackedFunc), and the overhead is dominated by the calling to PackedFunc and related instructions such like alloc_tensor and load_const. Of course, we can do some work to reduce the number of auxiliary TVM Ops and VM Instructions, but ideally, the AOT approach should be more appealing, cause we can inline the small pieces of cpu calculation and eliminate the overhead of VM instructions, and the performance roofline should be higher than VM’s approach.