[RFC] Update RFC Process

Development pre-RFC
#1

Current Status

The current RFC process is nominally documented within the TVM Community Committer Guide. The current guide states that when planning a contribution, “after an in-person discussion, send a summary to public channels (as an RFC or a discuss thread).” The current process the community has adopted for RFCs is to post a thread to the TVM Discuss forums, engage in community discussion, reach lazy consensus, and possibly file a tracking issue.

Proposed Solution

The RFC proposes to formalize the RFC process, making each step of the process and desired outcome clear. The RFC motivation and process is described below. The process is intended to provide a consistent mechanism for introducing new features and major changes to TVM, giving all stakeholders a clear and confident view of the direction TVM is evolving in.

What is an RFC?

An RFC is a “Request for Change” to the TVM project. It is a design document that describes a new feature, enhancement, or process to the TVM project. RFCs should be the primary mechanism for proposing major features and changes. The author of the RFC is responsible for the discussion of the change, and for organizing the work around it. RFCs are text files that serve as history and documentation of features.

Who is the audience for RFCs?

The primary audience of RFCs is the TVM development community. They serve as a guide for the design and implementation of features during and after their development. A secondary audience is general users and developers who are interested in how and why a feature was designed and implemented.

RFC Workflow

  • Discussion
    • During the discussion phase, a need or issue is brought to the discussion forum. During this phase, the developer and user community can discuss the need for and requirements of the RFC
  • Pull Request
    • After or concurrent with the conversation on the discussion forum, a pull request is sent up with the formal RFC. The RFC should include the authors, current project status, proposed solution, timeline for delivery, and criteria for completion. Discussion about the details of the RFC can continue in the pull request. There are two options for how to manage official RFS in the git repository:
      • RO: The PR shall be made against the official Apache TVM repository against the docs/rfc/active directory.
      • R1: The PR shall be made against a new RFC repository for TVM.
    • A committer of the corresponding area will approve and merge the RFC. Normally the corresponding committer will become the shepherd of the implementation PRs.
    • RFCs are numbered consecutively based on their order of proposal, regardless of if they are accepted or postponed.
    • RFCs should follow the format specified by the template RFC.
    • A successful RFC will include an overview with the problem the RFC is attempting to address, a proposed solution that describes the design and implementation strategy, and a timeline for completion. Optional sections can include (but are not limited to) alternatives that were considered, security considerations, and open problems that the RFC does not solve.
    • It is expected that RFCs will change, as part of the feedback process and as new implementation details arise. Changes to the RFC should not be squashed or force pushed in order to retain change and discussion history.
  • Tracking Issue
    • Upon merging a RFC, a tracking issue will be created where implementors can continue sharing implementation details (including links to pull requests). The issue will be closed when the RFC is either completed or abandoned.
  • Implementation
    • Work will begin on the RFC, with pull requests linking back to the tracking issue. Upon completion of the RFC, the tracking issue will be closed and the RFC will be moved to the docs/rfc/completed directory.
    • Postponement
      • An RFC may be postponed either explicitly by the parties responsible for implementing it, or implicitly by having no work done for a period of time defined by project leaders. The RFC will be moved to the docs/rfc/postponed directory.
  • RFC Grooming
    • Committers with expertise that covers an RFC may opt toRFC grooming session will be announced on the Discuss forums. Committers can active RFCs and determine if they need to be postponed. This can include reaching out to RFC proposers to check on the status of an RFC. A committer postponed RFC will be moved to the docs/rfc/postponed directory.
  • Resuming an Postponed RFC
    • Work on an abandoned RFC may be resumed by a new responsible party at any time after another discussion and pull request review process to move the RFC from docs/rfc/abandoned to docs/rfc/active.
  • Documentation
    • RFCs will be considered to be project documentation, and will be indexed as part of the TVM docs website. RFCs should be formatted using Markdown to aid in automated documentation generation.

Implementation

The implementation details for the new RFC process are to:

  • Create an RFC template.
  • Create the RFC directory or repository.
  • Migrate the backlog of existing RFCs in Discuss to the repository, focusing on active RFCs and RFCs of note and create corresponding tracking issues.
  • Document the RFC process in the TVM developer docs.

Items for Discussion

  • How do we want to store RFCs?
    • R0: store in the same same repo:
      • apache/tvm in folder: docs/rfcs
    • R1: separate repo:
      • apache/tvm-rfcs
  • Do we want to identify outstanding RFCs in the forums and migrate them to the repository?
  • Is the process formal enough? Is it too formal?
2 Likes
[RFC] MLIR Frontend
#2

@hogepodge thanks for the RFC and it overall looks promising :slight_smile:

My two cents:

  • Does this process come up with a person to manage/tracking the RFCs? Since it involves so many stages, I’m afraid that this process is not guaranteed to work well if we only rely on committers/PPMC.

  • One current issue of posting RFC in the discuss forum is lacking attentions and feedback. Since most RFCs focus on a feature in a certain module, normally only the people working on that module could provide valuable feedback. However, it’s not guaranteed to hear from them in time. Could this process more or less solve this problem?

  • If the author has to file an RFC pull request and get approved/merged, it seems redundant to post the RFC to the discuss forum, because where ever we are going to maintain RFCs, that place must be public as well.

  • According to the previous point along with my personal opinion of where to store RFCs, I would suggest using the Github issues on the official TVM repo for the following reasons.

    1. Open/Comment/Modify an issue won’t trigger CI. This is also the reason that we should not store the RFC in the official TVM repo, because the CI loading is already too heavy.
    2. We can use labels to represent the RFC status (e.g., discussing, implementing, grooming, documenting, etc). If we use pull requests, it only has two states (open/close), and we still need to use labels if we want more.

  • I didn’t understand this statement: Do we want to identify outstanding RFCs in the forums and migrate them to the repository? Could you clarify a bit?

  • Is the process formal enough? Is it too formal? I like this process in general but you’re right. Sometimes it might be too formal for relatively small RFCs to be processed efficiently. On the other hand, it seems to me that this is the question about how we judge if a change should have an RFC first or not. Now we have a label “need RFC” for PRs that cannot get enough agreements. It might be good to apply the same idea to RFC status like “no need RFC”. If a certain number of people consider this is straightforward and the PR can be filed directly, we could bypass the rest process.

#3

Thanks for the feedback on this. To address your points in order:

  • My original draft had called for project maintainers to handle RFCs, and even called for a period of time when RFCs would be declared “postponed”. I don’t know if I have a good solution to this. I did lay out a process with a lot of steps, but I was trying to err on the side of being complete rather than leaving too much open to interpretation. It’s definitely a risk, and possibly one that we could mitigate as we add more reivewers and contributors. Part of leadership in the project could include being active in RFCs. I think it’s a good point for discussion.
  • This is one of the big points I’m trying to address. As RFCs are brought up in the the Discuss forum, it’s very easy for them to scroll off the front page. This proposal is meant to capture the RFCs, and put them into a place where they are discoverable and more actionable (including feedback).
  • This is another difficult one. The way I like to think of it, proposals are brought up in the Discuss forum to gain a general consensus, then are formalized in GitHub. It’s the evolution of “I have a good idea” to “we think this is a good idea, let’s make a permanent and discoverable record of it”.
  • My concern with this point is in RFC discovery and generation of documentation. Issues are great for tracking things that go away, but a well written RFC becomes a permanent initial document of a feature. To the sub-points, it would be interesting to know if there’s a way to mark some directories as not triggering CI jobs. This is the main reason a separate repository was listed as an alternative, to allow it to be independent of CI.
  • Let me restate the question: We have a number of existing RFCs in the Discuss forum. Is it worthwhile to create formal RFCs to document them in the upstream repository?
  • Makes sense.

Thanks for your feedback and questions @comaniac. This is really helpful.

#4

Thanks for the clarification and they make sense to me.

For how to reduce the risk of having those process steps, maybe we could consider some GitHub bots that keep tracking the status and post reminders for idle cases. I feel it could work well, although I don’t have experience with it except for the bot that posts the code coverage report…

For the process of posting on the forum first, it’s reasonable and we might let authors decide. Like “we have no clue if the community would like this idea so let’s try the forum first”.

For using the GitHub repo instead of issues, I agree with you if you expect the RFC proposals are written in a more formal format (e.g., RST) instead of just Markdown. I guess it’s also possible to exclude a directory from triggering the CI.

For existing RFCs, I think after we have the new place for RFCs, we can comment on existing RFCs in the forum and encourage authors to post the RFC there to start the process.

#5

The use of a separate repo apache/tvm-rfcs will likely resolve the issue of CI, as well as possibility of storing binary data such as images that we do not want to include as part of the release.

1 Like
#6

one thing i’m wondering–often times you need to write a PoC in order to write an RFC. I think the typical use case might be something more like:

  1. Discussion: forum discussion, create RFC PR, and include PoC in PR message history. wait for community consensus.
  2. RFC PR: now, commit the RFC PR to e.g. tvm-rfcs and create a tracking issue
  3. Implementation: polish PoC and split into PRs, each of which mention the tracking issue

does that still fit with this process? my only concern is that it might not be possible to have a good discussion about an RFC that affects many parts of the code if some PoC hasn’t been made yet.

on where accepted RFCs belong: I do feel that we should serve them on the TVM docs site. My main concern is discoverability, and I’m not 100% sure how discoverable markdown files are in GH repos. we could do this either by including them in the original TVM repo or with e.g. GitHub Pages. it seems easier to put them in the original repo, but we do have to address the CI bit. i am hoping this quarter to change the CI requirements for docs/ only changes to a) build GPU b) generate docs; this may address that problem.

#7

Different part of the tvm-site can be built with different means, given that we are already using two different mechanisms to build the main site and the docs, we could add another script to pull and serve the rfc docs under tvm.apache.org/rfcs

#8

One thing I don’t want is for a process to work against the way the community needs. Including a PoC component sounds like an essential part of this.

#9

I think another repo e.g. apache/tvm-rfcs is good, so that it doesn’t create noise in the source/documentation repo. Using the PRs is also good because it shows all the comments in place, so the comments in a particular section don’t get lost in the discussion.

In GitHub, we can cross-link PRs and issues, so there is nothing the prevents us of having PoC code running fine with CI in the main repo apache/tvm, and the RFC discussion going on in the dedicated repo.

I think it is formal enough, and is a positive move.

#10

It seems like we have reached consensus on moving to a repo to store RFCs, what process do we need to kick off in order to create such a repo? I would like to start submitting some new RFCs I will write to the new process.

#11

repo created apache/tvm-rfcs · GitHub. We might need some intial effort on setting up the initial structure, then we can create an announcement to start the new process moving forward. @jroesch can you take a stab at the initial scafolding?

1 Like
#12

The RFC repository has been seeded with a template, and I’ve sent up a pull request that attempts to capture the process discussed here. Please take a look and review, and we can formally begin the RFC process once that is complete.

Thanks! Chris

#13

One item up for discussion in the pull request process is if we should link back to the original discussion forum postings for RFCs. The options are to:

A1: Link to the original posting to provide additional context, but treat the RFC as the source of truth for the RFC A2: Don’t link back to original posting, with a strong bias towards treating the RFC as the source of truth.

#14

Thanks for everyone’s discussion. Please participating the vote here [VOTE] Adopt the New RFC Process · Issue #7991 · apache/tvm · GitHub to formally adopt the RFC process

#15

The vote on the new RFC process has completed, with 18 votes to accept. In accordance with lazy consensus, the new process is in effect. As new RFCs are posted to the forum, I will add reminders to prepare a pull request to formalize the change. Thanks to everyone who provided feedback, and for helping to better organize and track this work through the new process.

#16

Hi @hogepodge ,

If we are to store images, what is directory structure to keep them? Im thinking whether we’d need a folder per rfc ?

#17

We can always hold images and other assets in a different folder, or a different location

#18

@tqchen ,

Yes, we could. However, how would we associate the the group of assets with a rfc ? Maybe an assets/(rfc PR)/ ?

(Im assuming the first number in the rfc .md is the PR number, please correct me if I am wrong)

#19

My personal feeling is that we should hold assets in the same repository, but we should also make sure those assets are reasonably sized.

For discussion, I propose a max 1MB/image.

The issue we can run into with images is the repository could quickly become unreasonable to work with. The issue with hosting externally is that you have to provide guarantees on long-term availability that you don’t have to worry about with in-repository images.

#20

I broadly agree with holding them in same repository.

My question was what should be the directory structure :slight_smile: Other option for images, we could maybe link them from the discuss?

E.g. : https://discuss.tvm.apache.org/uploads/default/original/2X/0/0c28cd11e625906bcec06bab91bee67f1eab278c.png