This is an important discussion to have. I think one big missing piece is approximate template matching. Right now if a workload doesn’t have an exact match in tophub, the default fallback configuration is used which of course leads to slow inference. However, workloads of similar shapes / type are highly likely to share the same optimal schedule. It would make sense to introduce some way to recommend the most likely good schedule for a new workload instead of having to tune it from scratch. That would make the tophub predefined schedules apply to many new models.
3 Likes