[RFC] Update RFC Process

Development pre-RFC
#2

@hogepodge thanks for the RFC and it overall looks promising :slight_smile:

My two cents:

  • Does this process come up with a person to manage/tracking the RFCs? Since it involves so many stages, I’m afraid that this process is not guaranteed to work well if we only rely on committers/PPMC.

  • One current issue of posting RFC in the discuss forum is lacking attentions and feedback. Since most RFCs focus on a feature in a certain module, normally only the people working on that module could provide valuable feedback. However, it’s not guaranteed to hear from them in time. Could this process more or less solve this problem?

  • If the author has to file an RFC pull request and get approved/merged, it seems redundant to post the RFC to the discuss forum, because where ever we are going to maintain RFCs, that place must be public as well.

  • According to the previous point along with my personal opinion of where to store RFCs, I would suggest using the Github issues on the official TVM repo for the following reasons.

    1. Open/Comment/Modify an issue won’t trigger CI. This is also the reason that we should not store the RFC in the official TVM repo, because the CI loading is already too heavy.
    2. We can use labels to represent the RFC status (e.g., discussing, implementing, grooming, documenting, etc). If we use pull requests, it only has two states (open/close), and we still need to use labels if we want more.

  • I didn’t understand this statement: Do we want to identify outstanding RFCs in the forums and migrate them to the repository? Could you clarify a bit?

  • Is the process formal enough? Is it too formal? I like this process in general but you’re right. Sometimes it might be too formal for relatively small RFCs to be processed efficiently. On the other hand, it seems to me that this is the question about how we judge if a change should have an RFC first or not. Now we have a label “need RFC” for PRs that cannot get enough agreements. It might be good to apply the same idea to RFC status like “no need RFC”. If a certain number of people consider this is straightforward and the PR can be filed directly, we could bypass the rest process.

#3

Thanks for the feedback on this. To address your points in order:

  • My original draft had called for project maintainers to handle RFCs, and even called for a period of time when RFCs would be declared “postponed”. I don’t know if I have a good solution to this. I did lay out a process with a lot of steps, but I was trying to err on the side of being complete rather than leaving too much open to interpretation. It’s definitely a risk, and possibly one that we could mitigate as we add more reivewers and contributors. Part of leadership in the project could include being active in RFCs. I think it’s a good point for discussion.
  • This is one of the big points I’m trying to address. As RFCs are brought up in the the Discuss forum, it’s very easy for them to scroll off the front page. This proposal is meant to capture the RFCs, and put them into a place where they are discoverable and more actionable (including feedback).
  • This is another difficult one. The way I like to think of it, proposals are brought up in the Discuss forum to gain a general consensus, then are formalized in GitHub. It’s the evolution of “I have a good idea” to “we think this is a good idea, let’s make a permanent and discoverable record of it”.
  • My concern with this point is in RFC discovery and generation of documentation. Issues are great for tracking things that go away, but a well written RFC becomes a permanent initial document of a feature. To the sub-points, it would be interesting to know if there’s a way to mark some directories as not triggering CI jobs. This is the main reason a separate repository was listed as an alternative, to allow it to be independent of CI.
  • Let me restate the question: We have a number of existing RFCs in the Discuss forum. Is it worthwhile to create formal RFCs to document them in the upstream repository?
  • Makes sense.

Thanks for your feedback and questions @comaniac. This is really helpful.

#4

Thanks for the clarification and they make sense to me.

For how to reduce the risk of having those process steps, maybe we could consider some GitHub bots that keep tracking the status and post reminders for idle cases. I feel it could work well, although I don’t have experience with it except for the bot that posts the code coverage report…

For the process of posting on the forum first, it’s reasonable and we might let authors decide. Like “we have no clue if the community would like this idea so let’s try the forum first”.

For using the GitHub repo instead of issues, I agree with you if you expect the RFC proposals are written in a more formal format (e.g., RST) instead of just Markdown. I guess it’s also possible to exclude a directory from triggering the CI.

For existing RFCs, I think after we have the new place for RFCs, we can comment on existing RFCs in the forum and encourage authors to post the RFC there to start the process.

#5

The use of a separate repo apache/tvm-rfcs will likely resolve the issue of CI, as well as possibility of storing binary data such as images that we do not want to include as part of the release.

1 Like
#6

one thing i’m wondering–often times you need to write a PoC in order to write an RFC. I think the typical use case might be something more like:

  1. Discussion: forum discussion, create RFC PR, and include PoC in PR message history. wait for community consensus.
  2. RFC PR: now, commit the RFC PR to e.g. tvm-rfcs and create a tracking issue
  3. Implementation: polish PoC and split into PRs, each of which mention the tracking issue

does that still fit with this process? my only concern is that it might not be possible to have a good discussion about an RFC that affects many parts of the code if some PoC hasn’t been made yet.

on where accepted RFCs belong: I do feel that we should serve them on the TVM docs site. My main concern is discoverability, and I’m not 100% sure how discoverable markdown files are in GH repos. we could do this either by including them in the original TVM repo or with e.g. GitHub Pages. it seems easier to put them in the original repo, but we do have to address the CI bit. i am hoping this quarter to change the CI requirements for docs/ only changes to a) build GPU b) generate docs; this may address that problem.

#7

Different part of the tvm-site can be built with different means, given that we are already using two different mechanisms to build the main site and the docs, we could add another script to pull and serve the rfc docs under tvm.apache.org/rfcs

#8

One thing I don’t want is for a process to work against the way the community needs. Including a PoC component sounds like an essential part of this.

#9

I think another repo e.g. apache/tvm-rfcs is good, so that it doesn’t create noise in the source/documentation repo. Using the PRs is also good because it shows all the comments in place, so the comments in a particular section don’t get lost in the discussion.

In GitHub, we can cross-link PRs and issues, so there is nothing the prevents us of having PoC code running fine with CI in the main repo apache/tvm, and the RFC discussion going on in the dedicated repo.

I think it is formal enough, and is a positive move.

#10

It seems like we have reached consensus on moving to a repo to store RFCs, what process do we need to kick off in order to create such a repo? I would like to start submitting some new RFCs I will write to the new process.

#11

repo created apache/tvm-rfcs · GitHub. We might need some intial effort on setting up the initial structure, then we can create an announcement to start the new process moving forward. @jroesch can you take a stab at the initial scafolding?

1 Like
#12

The RFC repository has been seeded with a template, and I’ve sent up a pull request that attempts to capture the process discussed here. Please take a look and review, and we can formally begin the RFC process once that is complete.

Thanks! Chris

#13

One item up for discussion in the pull request process is if we should link back to the original discussion forum postings for RFCs. The options are to:

A1: Link to the original posting to provide additional context, but treat the RFC as the source of truth for the RFC A2: Don’t link back to original posting, with a strong bias towards treating the RFC as the source of truth.

#14

Thanks for everyone’s discussion. Please participating the vote here [VOTE] Adopt the New RFC Process · Issue #7991 · apache/tvm · GitHub to formally adopt the RFC process

#15

The vote on the new RFC process has completed, with 18 votes to accept. In accordance with lazy consensus, the new process is in effect. As new RFCs are posted to the forum, I will add reminders to prepare a pull request to formalize the change. Thanks to everyone who provided feedback, and for helping to better organize and track this work through the new process.

#16

Hi @hogepodge ,

If we are to store images, what is directory structure to keep them? Im thinking whether we’d need a folder per rfc ?

#17

We can always hold images and other assets in a different folder, or a different location

#18

@tqchen ,

Yes, we could. However, how would we associate the the group of assets with a rfc ? Maybe an assets/(rfc PR)/ ?

(Im assuming the first number in the rfc .md is the PR number, please correct me if I am wrong)

#19

My personal feeling is that we should hold assets in the same repository, but we should also make sure those assets are reasonably sized.

For discussion, I propose a max 1MB/image.

The issue we can run into with images is the repository could quickly become unreasonable to work with. The issue with hosting externally is that you have to provide guarantees on long-term availability that you don’t have to worry about with in-repository images.

#20

I broadly agree with holding them in same repository.

My question was what should be the directory structure :slight_smile: Other option for images, we could maybe link them from the discuss?

E.g. : https://discuss.tvm.apache.org/uploads/default/original/2X/0/0c28cd11e625906bcec06bab91bee67f1eab278c.png

#21

I like the idea of an assets subdirectory, with further subdirectories named by RFC number.