[pre-RFC] TVM Roadmap

:clapper: Note: This pre-RFC was presented at the September TVM Community Meeting (check out the video here). Looking forward to gathering more community feedback!

Summary

This RFC proposes to add product roadmaps to TVM.

Roadmaps should be seen as a way of unifying the planning process of TVM, all the way from ideas to PR merges. The roadmaps discussed in this pre-RFC are intentionally designed to integrate with TVM’s existing planning tools (e.g. Github tracking issues, RFCs), while adding an additional space for sharing and collaboration earlier in the R&D process.

This proposal includes two categories of roadmaps:

  • TVM Global Roadmap: This roadmap aims to provide a holistic view of all components within TVM, focusing on features which broadly affect the entire project or add significant functionality.
  • TVM Component Roadmaps: These roadmaps aim to provide a more detailed view of a subsection of TVM: including vertical efforts involving TVM’s intermediate representations (such as Relay or TIR) and the horizontal efforts which span across TVM’s full stack (such as Automation or Documentation). Details about each of the components in TVM will be discussed later in this pre-RFC.

Motivations

  • Integrate TVM’s technical vision and task tracking mechanisms together
  • Provide everyone with visibility into all of TVM’s projects
  • Encourage collaborative and open planning in the TVM community

Guide-Level Explanation

This guide-level explanation will describe the basic structure of the roadmap, without focusing heavily on the intended contents.

Tooling

The roadmap utilizes Github Projects as the underlying tooling mechanism. This maximizes reuse of existing content tracked in Github, and is very user-friendly for existing Github users.

This RFC proposes to place the TVM roadmaps directly in the apache/tvm repository, since the roadmap is intended to be a community-focused project, and having the roadmaps directly in TVM would help to maximize visibility to the TVM community.

Sample Roadmap

Above is a sample roadmap for TVM CI and Testing, with annotations to show several important features within the roadmap:

  1. Background & Motivations Column: This section is located on the left-most part of the roadmap, and is intended to define a few key focus areas of the project and their overall success criteria. In the TVM CI and Testing roadmap, this section contains definitions of Coverage, Ease of Use, Runtime, Stability, and Reporting.
  2. Quarterly Planning Columns: Following Background & Motivations, a roadmap has 4 quarterly planning columns (one column per upcoming quarter through the next year). Each Quarterly planning column has these cards:
    • Themes & Goals: The Themes & Goals are listed in the top-most card of each column of the roadmap, and are intended to show how the focus areas of the project are addressed that quarter.

      • Themes are chosen from those defined in the Background & Motivations section (such as Stability or Runtime).
      • Goals describe how the Roadmap Items listed in each column contribute to the success criteria listed in Background & Motivations.

      For example, in Q3 of 2021 (the last 3 months of development in TVM), two of the major focus areas in TVM CI and Testing were Stability and Runtime.

    • Roadmap Items: Each column of development contains roadmap items which contribute to each of the Themes & Goals described above. These items are meant to unify feature development within each project, whether it is coming from an RFC, Github task tracking, or directly from within the roadmap. Items on the roadmap generally have feature ownership at least partially identified at a minimum.

  3. Backlog Column: Items on the roadmap which don’t have an owner or a proposed development timeline fall into the Backlog Items column. This is also the default location where new roadmap issues are placed.

Reference-Level Explanation

Design Considerations

There were several key considerations made while designing this roadmap, centering around the overall themes of quality content, usability, and maintainability. Below are some of these considerations:

  1. Make it as easy as possible for contributors without domain-level expertise to quickly gain basic context and understanding of a project
  2. Encourage consistent and high-quality roadmap content
    • Background Information
      • There should be a roadmap for all of TVM, and detailed, focused roadmaps for components of TVM
      • Each roadmap should contain themes which encapsulate key focus areas of the roadmap
      • Each roadmap should contain items which describe either features or milestones
        • Each item should have clearly defined motivations, end goals, deliverables, and scope
        • Smaller tasks (e.g. fixing flaky tests) may be combined into a larger milestone
        • Larger features (e.g. AutoTIR) may be split into smaller milestones
    • Schedule & Prioritization
      • Priority of any given feature should be determined by consensus, then displayed prominently
      • Dependencies and blockers should be easily traceable
      • Delivery schedule should be estimated based on prioritization, dependencies, and blockers, and it should be updated as confidence increases
    • Ownership
      • Subject-matter experts (e.g. feature authors, contributors with subject-matter expertise) should be listed for each feature
      • Task-level ownership should be listed for each feature
      • Tasks with help needed should be discoverable to new users
  3. Leverage existing task tracking mechanisms
    • Easier maintenance for everyday contributors
    • Show updated progress towards feature completion
  4. Encourage sharing and collaboration in the early stages of planning
    • Roadmap describes what changes are desired
    • RFC describes how changes will be implemented

Proposed Roadmap Structure

Below is the initial set of roadmaps proposed in this RFC.

Roadmap Name Scope
TVM Global Roadmap All of TVM’s efforts
TVM Automation Roadmap Automated performance improvements across the entire TVM stack
TVM Community Roadmap Community efforts across all of TVM, including Developer Tooling, Documentation, Events, Videos, and Tutorials
TVM Continuous Integration & Testing Roadmap CI and testing across all of TVM
TVM Core Compiler Roadmap Major compiler features and/or refactors with impacts across key APIs and/or multiple IRs in the TVM stack
TVM Frontend & Interface Roadmap Major user-facing pieces of TVM, such as importers, TVMC, and user facing APIs
TVM Graph Computation & High-Level Optimizations Roadmap All things Relay related
TVM Scheduler & Low-Level Optimizations Roadmap All things TIR related
TVM Hardware Roadmap All things hardware backend related
microTVM roadmap All things microTVM related

Challenges

There are a few challenges to having publicly available roadmaps, but overall, we believe that the benefits of having roadmaps for TVM vastly outweigh the negatives.

  • Changes to the roadmap: Items on the roadmap are subject to change as plans and community consensus changes. It is inevitable that projects get delayed or rescoped with time, and this will impact the roadmaps. However, the true value of roadmapping is to empower contributors to share their ideas, changes included, with the community.
  • Contents of the roadmap: TVM has a broad and diverse contributor base who may take different approaches to authoring and reading the contents of the roadmap. In order to ensure uniformity and high-quality contents within the roadmap, community roadmap guidelines will be created and upstreamed.
  • Organization of the roadmap: Since the proposed roadmaps would live directly in apache/tvm via Github Projects, roadmap items would be created directly in apache/tvm as Github issues. Currently, Github issue usage is very limited in apache/tvm in order to ensure easier triage and timely closure of issues. Having the TVM roadmaps live in apache/tvm would increase the overall count of Github issues in the repository, but the roadmap items could be labeled with a tvm:roadmap tag so that they can be properly triaged and filtered.

Rationale & Alternatives

There are several alternatives to this roadmap proposal; however, each of the alternatives has drawbacks of its own.

  • Do nothing: Don’t create a publicly viewable roadmap. However, the community would miss out on the numerous benefits of having a roadmap.
  • Create a separate repository for roadmap: Having a separate roadmap repository (e.g. apache/tvm-roadmap) might reduce the number of roadmap issues filed directly to apache/tvm, but it would increase the complexity and fragmentation of the roadmap significantly, since this roadmap repository would have to link to RFC tracking issues in apache/tvm anyways.
  • Use a separate tool for the roadmap: Using a separate tool for the roadmap may provide more functionality than Github projects. However, this would require much more effort to create and maintain.

Prior Art

This RFC is inspired by the previous work done to improve the tracking mechanisms of TVM. This includes but is not limited to the existing RFC process, release process, and technical visions of TVM.

It is also inspired by the public Github roadmap, which is an excellent example of usage for Github Projects.

Upstreaming Timeline

This is a prospective upstreaming timeline for the TVM roadmap.

Milestone Name Deliverables Schedule
TVM Roadmap M1A: Taking the Plunge
  • All roadmap projects defined
  • Initial prototype roadmap (TVM CI & Testing) upstreamed
  • Skeleton roadmap documentation upstreamed
  • Scope of M1B and M1C clearly defined
  • Late September
    TVM Roadmap M1B: Building Momentum
  • At least 3 additional roadmaps upstreamed
  • Additional roadmap process documentation upstreamed
  • Mid October
    TVM Roadmap M1C: Ready for Prime Time
  • All initial roadmaps (described in M1A RFC) upstreamed
  • All RFC opens from M1A and M1B addressed
  • Publicize roadmap in documentation and TVM forums
  • Early November

    Unresolved Questions

    Questions about the design and contents of the TVM Roadmap will be progressively resolved through the upstreaming timeline shown above. Any unresolved questions at the end of TVM Roadmap Milestone 1 will be created as roadmap items in the TVM Community Roadmap :slightly_smiling_face:

    Future Possibilities

    Special Thanks

    Thanks so much for copyediting, @areusch and @electriclilies! :hugs:

    15 Likes

    thanks @denise-k for your proposal! Since I helped with editing the proposal here I of course think it’s a great starting point. We would love to hear more feedback from the community, other committers, and PMC!

    In particular for microTVM I would propose we move away from the periodic M Roadmap threads and onto something like this, which provides more frequent updates and continuity.

    ccing a few others @tqchen @jroesch @masahi @junrushao1994 @thierry @vinx13 @FrozenGene @jwfromm @comaniac @yzhliu @zhiics @ramana-arm @kparzysz @aca88

    1 Like

    This is a fantastic move in our TVM community. Super excited! :wink:

    As a distributed community, I would say different developers from different companies/universities/etc may like to collaboratively working on items in the roadmap, or adding projects to it. Therefore, the community roadmap guidelines would be really helpful to make the process really smooth. Looking forward to it!

    1 Like

    Yes! Absolutely agree that we should make the roadmap a collaborative process :raised_hands: Looking forward to releasing documentation outlining this in M1B.

    1 Like

    Thank you @denise for steering our community towards having a clearer roadmap. Looking forward to seeing this collaborative process flourish and obtaining input and suggestions from the rest of the community!

    1 Like

    Overall would love to move to a larger and more encompassing roadmap so we can communicate across companies, universities and individuals what we are working on and the general status. :+1: :+1:

    2 Likes

    Thanks @denise for proposing this and looking forward to the roll out

    1 Like