Updated Docs pre-RFC

Development pre-RFC
#8

Just read through this and providing my own opinions. I’m a huge fan of L2 - Tour Style here, and I appreciate that it blends topics such as TVM and microTVM in the beginning rather than treating them as separate; it makes a lot of sense to me to use this to ensure we make all of TVM approachable and as exposed as possible to potential users. As such the starting point of a quick Tutorial and How Tos for core user journeys makes a lot of sense to minimise friction. Keeping a clear divide is important here, as the core Tutorial provides the introduction whilst the How Tos provide “ok here’s your use case and what buttons to press” (I believe this is Q0a + Q1a).

As a user following this my next point of call would be to try and find a matching How To to a more exact use case, or decide to read some Explanation of that concept - which I believe the Architectural Guide does well (Q3b?).

I think this is essentially:

  • Getting Started
    • Tutorial
    • How To
  • Tutorials
    • Tutorial
    • How To
  • Specific Things
    • Topic
      • Explanation
      • How To
      • Reference

This collects Topics under the Specific Things section so you don’t have two places to look to find details on a specific topic - at the cost of one level of nesting.

I understand your sentiment here, but I would expect that divide to occur as efforts like this make TVM more approach-able and I would therefore plan for TVM being usable by a broad spectrum of users.

Following from the above, I’d question whether this is actually how we want to consider users, once the groundwork is in I would expect users to simply run tvmc or a python script against a number of developer created backends similar to how a user would use any other ML framework (without knowing the intricacies of its operation). I would agree with keeping the docs together though, the L2 - Tour Style would then let you go as deep as you need to get your model optimised.

Overall, I’m in favour of the formalisation to allow us to have discussions regarding placement and I think this provides a huge improvement in clarity.

#9

Thanks @hogepodge for the RFC. Personally, I prefer L2 style and I think it is the easiest way to dive into TVM world. From my point of view, this style is very friendly for new users, but in the same time it is well organized, and it helps to find useful information in short time for experienced users.

Also, I think it is necessary to have well organized API reference documentation for developers. Because sometimes it is not necessary to read long documentation, but you need to find information about a function in faster way and use it. I really like how OpenCV documentation is organized. You can read tutorials about different components and modules, but at the same time you can use search to find documentation about any function and read about it. Also, for some functions, there are reference links in the description to tutorials where you can see how this function should be used.

I think, this is a good example how API documentation can be useful for users and developers at the same time. And I think that API documentation should contain not only Python part but also different internal C++ functions.

One important thing which should be included in the documentation is the current status for different frontends/backends/runtimes etc. Because sometimes we don’t know the current status of frontend/backend/runtime, and we should try to use it before understanding is it workable/usable or not. I have one real example of such situation. One my friend tried to use TVM in his own project. He used network based on YOLOv4 and used darknet frontend to convert the original model to TVM. But on the inference, he got inaccurate results. And he was able to solve this problem only by converting original darknet model to ONNX format, and then he used ONNX frontend. After these exercises, the inference gave the necessary accuracy. Also, I personally many times meet some issues in PyTorch frontend for example and only after converting the original model to ONNX format everything was fine. I suppose that it would be nice to have a good documentation about each frontend/backend/runtime etc. with specifying current status and limitations.

#10

Reading through the comments, I want to clarify on the options presented. I had copy-and-pasted L2 from a previous suggestion, and it included more detail than L0 or L1. My concern with L2 is that it actually lacks structure, and is more similar to the current situation than a proposed improvement. Additionally it doesn’t follow the terminology as outlined in the proposal.

For example, what are demos? Are they how-tos? Tutorials? Here is expanded set of options with more details, based on existing documents reorganized into this structure.

L0/L1 Refined, expanded

  • Getting Started
    • About TVM
    • Installing TVM
  • User Guide
    • Tutorial
      • Introduction
      • An Overview of TVM and Model Optimization
      • Installing TVM
      • Compiling and Optimizing a Model with TVMC
      • Compiling and Optimizing a Model with the Python Interface (AutoTVM)
      • Working with Operators Using Tensor Expression
      • Optimizing Operators with Schedule Templates and AutoTVM
      • Optimizing Operators with Auto-scheduling
      • Cross Compilation and RPC
      • Introduction to TOPI
      • Quick Start Tutorial for Compiling Deep Learning Models
    • How To
      • Install TVM
        • Install from Source
        • Docker Images
      • Compile Deep Learning Models
      • Deploy Deep Learning Models
      • Work With Relay
      • Work with Tensor Expression and Schedules
      • Optimize Tensor Operators
      • Auto-Tune with Templates and AutoTVM
      • Use AutoScheduler for Template-Free Auto Scheduling
      • Work With microTVM
  • Topic Guides
    • MicroTVM Guide (index to existing docs)
      • → Work With microTVM
      • → microTVM Architecture
    • VTA (index to existing docs)
  • Developer Guide
    • Tutorial: Contributing to TVM
      • TVM Community Guideline
      • Performing Code Reviews
      • Committer Guide
      • Writing Document and Tutorials
      • Code Guide and Tips
      • Error Handling Guide
      • Submitting a Pull Request
      • Git Usage Tips
      • Apache TVM Release Process
    • How Tos
      • Write an operator
      • Write a backend
      • …
  • TVM Architecture Guide (deep dive)
    • Architecture Overview
    • Research Papers
    • Frontend
    • Relay: Graph-level design: IR, pass, lowering
    • TensorIR: Operator-level design: IR, schedule, pass, lowering
    • TOPI: Pre-defined operators operator coverage
    • AutoScheduler / AutoTVM: Performance tuning design
    • Runtime & microTVM design
    • Customization with vendor libraries BYOC workflow
    • RPC system
    • Target system
  • API Reference (reference)
    • Generated C++ Docs…
    • Generated Python Docs…

**L2 - to match terminology **

When you modify L2 to match the terminology introduced above, it looks more like L0/L1, but to my eyes is a bit more confusing to follow along with.

  • Get started
    • Installating TVM
    • How to: Optimize an operator
    • How to: Optimize a neural net (w/ TVMC, w/o TVMC)
    • How to: Minimal deployment (w/ TVM runtime, w/ microTVM runtime)
    • How to: Contribute
  • How Tos
    • How to use Relay to Import to Relay example: LeNet-5 on MNIST
    • How to use Relay to Build Relay to binary example: LeNet-5 on MNIST
    • TE: Implement operators in Tensor Expression (compute & schedule) example: matrix multiplication
    • AutoScheduler / AutoTVM: Tune operators example: matrix multiplication
    • microTVM: deployment
    • mobile deployment
  • Architectural Guide
    • Architecture Overview
    • Research Papers
    • Frontend
    • Relay: Graph-level design: IR, pass, lowering
    • TensorIR: Operator-level design: IR, schedule, pass, lowering
    • TOPI: Pre-defined operators operator coverage
    • AutoScheduler / AutoTVM: Performance tuning design
    • Runtime & microTVM design
    • Customization with vendor libraries BYOC workflow
    • RPC system
    • Target system
    • …
  • Topics of interests
    • microTVM
    • VTA
    • …
  • More How Tos
    • How to write Sparse models
    • …
#11

Thanks @hogepodge for starting the discussion. In my view, I really like the idea of having the option of having a tour style organization as a way to provide a new comer a clear thread to navigate documentation. However I get how L2 could end up making the structure a little less structured at the end of the day (e.g. “having a more how tos section”). All in all, I think the proposed L1 or L0 would be perfectly adequate to organize our documentation. I’d add that it may be helpful to offer a user-journey focused thread that a new comer could use to get in an afternoon exposure to a good collection of documents.

#12

@thierry For L2, I’m having a really hard time figuring out how to make contributions repeatable and describable. If you could expand on how the formalism behind it could work, that would be helpful to me.

#13

Thanks @hogepodge . A minor suggestion

It would be great to leave the contributor guideline in a top-level discoverable place(how about as part of get started? or top level), since the contributor guide are something that we frequently refers to

#14

Fair point, I agree this is the main flaw of L2. The point I wanted to make is that we can aim to have contributions be repeatable and describable on by adopting the L0 or L1 organization. A tour of TVM is possible by having a possible top level “tour page” that links to a sample of resources to make TVM approachable to a new comer. The tour is something that can require higher degrees of curation - I agree that this is something that may not be easy to self-maintain. So really that’s an orthogonal question to overall organization of documentation given that it can be prepared in many different forms.

1 Like
#15

For a project as complex as TVM, I think the separation of User and Developer content is really important to give new people a chance at understanding what is going on and how they can leverage TVM. I think the comments on this RFC are going to be a bit biased on this point as well, since most here are already sophisticated TVM developers, and from that perspective it’s hard to remember how overwhelming all of the concepts, content, and flexibility of TVM are.

#16

I’d lean toward L1 overall. I like the split between user guides and developer guide. Somebody who comes to TVM with an existing model to run and somebody who comes to TVM with a new piece of hardware are very different audiences, and should have different workflows.

I don’t like the split in API reference present in L0. If I’m looking at documentation and a function name doesn’t show up, my first assumption is that I’m looking at the wrong/outdate reference page, not that there was a split in the documentation.

My concern with L2 is that it while it would give the smoothest introduction to individually documented tasks, doing anything outside the explicitly documented paths becomes harder to find. Having these as how-tos, such as in @hogepodge’s expanded outline, lets there be a searchable URL, but doesn’t make them be the central feature.

#17

Speaking from experience with the TVM documentation, one of the challenges with the tutorials that exist today is that none of them fully cover many of my use cases, so one is forced to kind of cobble together something from multiple tutorials – rather than having a clear understanding of the fundamental capabilities. This can lead to “voodoo coding” to make something work rather than getting it from first principles. Also, the different tutorials tend to have a different voice, slightly different approach to doing similar things, and there is some amount of overlap between them. It’s hard to generalize.

I think what @hogepodge is proposing for L0/L1 addresses this by getting some of the overview out of the way up front, showing the general use case with TVMC, before diving into the lower-level details. I think this is still very much about providing users an easy onboarding path, but doing so in a way that does not limit them so much. From a user perspective I think the L0/L1 approach works much better and is more in line with other OSS projects. I think it provides a better foundation for expanding the docs over time as well.

1 Like
#18

Thanks everyone for great thoughts and discussions, reading the threads, I think there are two kind of motivations that goes into this.

  • M0: The need of a Formalism in terms of separating the things into clear categories
  • M1: The need to have a tour-style map to the beginners so they can navigate

a lot of us agreed on Q0a, separation of tutorial from how to that clearly splitting the structures. On the same line, I agree that the split of developer and user facing how tos are helpful.

While L2 captures M1. We can think about is how to effectively address M1 in a document system that have L0/L1 style.

Specifically, we might be able to tweak the Getting started section a bit to incorporate some of the elements guided map elements(e.g. like Q1a) but keeps the rest of the section as clear L0/L1 style separation.

#19

BTW, might be not strictly related to this thread, but could be good to bring it up: shall we have a workflow figure like this the one blow + plus some missing layers? (introduced in this RFC)?

image

We may link each component in the figure to their corresponding docs. For example, the logos on the first row link to importer docs; the second row to Relay, etc; The optimization column can be replaced with specific topics: VTA, microTVM, AutoScheduler, etc

1 Like
#20

I believe this would be a good navigation page for the architecture guide, and can fit into both L0/L1 and L2 style

#21

In reviewing the feedback, it appears that there is a preference for including easier access for some specialized pages, and ordering the layout in a way that moves through the user and developer journey of beginner to expert. I’ve refined the discussion into a new structure that attempts to address these issues. I’ve included three representations of it below to help give a high-level overview of what this would look like, but also dig into the specific details of where existing documents would land in this structure.

It creates placeholders for some new documents (such as the TVM Developer Tutorial and an Architecture Diagram/Map), but largely is focused on document organization and relocation of existing documents.

Within the Getting Started section there’s room for additional documents, such as a higher-level tour of TVM. This structure doesn’t preclude the creation of these sorts of documents, but rather creates a framework for their addition.

If this framework is generally acceptable for the community, we can begin writing a formal RFC for consideration.

L3 - Top Level

  • Getting Started
  • User Guide
  • TVM Topic Guide
  • Developer Guide
  • TVM Architecture Guide
  • API Reference (reference)
  • Index

L3 - Major Sections

  • Getting Started
    • About TVM
    • Installing TVM
    • TVM Contributor Guide
  • User Guide
    • Tutorial
    • How To
  • TVM Topic Guide
    • MicroTVM Guide (index to existing docs)
    • VTA (index to existing docs)
  • Developer Guide
    • TVM Contributor Tutorial (new, to be written)
    • How To
  • TVM Architecture Guide
    • Architecture Overview (new, diagram/map, to be written)
    • …
  • API Reference (reference)
    • Generated C++ Docs…
    • Generated Python Docs…
  • Index

L3 - Detailed

  • Getting Started
    • About TVM
    • Installing TVM
    • Contributor Guide
      • TVM Community Guideline
      • Performing Code Reviews
      • Committer Guide
      • Writing Document and Tutorials
      • Code Guide and Tips
      • Error Handling Guide
      • Submitting a Pull Request
      • Git Usage Tips
      • Apache TVM Release Process
  • User Guide
    • Tutorial
      • Introduction
      • An Overview of TVM and Model Optimization
      • Installing TVM
      • Compiling and Optimizing a Model with TVMC
      • Compiling and Optimizing a Model with the Python Interface (AutoTVM)
      • Working with Operators Using Tensor Expression
      • Optimizing Operators with Schedule Templates and AutoTVM
      • Optimizing Operators with Auto-scheduling
      • Cross Compilation and RPC
      • Introduction to TOPI
      • Quick Start Tutorial for Compiling Deep Learning Models
    • How To
      • Install TVM
        • Install from Source
        • Docker Images
      • Compile Deep Learning Models
      • Deploy Deep Learning Models
      • Work With Relay
      • Work with Tensor Expression and Schedules
      • Optimize Tensor Operators
      • Auto-Tune with Templates and AutoTVM
      • Use AutoScheduler for Template-Free Auto Scheduling
      • Work With microTVM
  • TVM Topic Guide
    • MicroTVM Guide (index to existing docs)
      • → Work With microTVM
      • → microTVM Architecture
    • VTA (index to existing docs)
  • Developer Guide
    • TVM Contributor Tutorial
      • …
    • How To
      • Write an operator
      • Write a backend
      • …
  • TVM Architecture Guide
    • Architecture Overview
    • Research Papers
    • Frontend
    • Relay: Graph-level design: IR, pass, lowering
    • TensorIR: Operator-level design: IR, schedule, pass, lowering
    • TOPI: Pre-defined operators operator coverage
    • AutoScheduler / AutoTVM: Performance tuning design
    • Runtime & microTVM design
    • Customization with vendor libraries BYOC workflow
    • RPC system
    • Target system
  • API Reference (reference)
    • Generated C++ Docs…
    • Generated Python Docs…
  • Index
#22

Thanks @hogepodge .

Placing higher-level tour into Getting started would indeed help us incorporate elements from L2 into this framework and combines the concern of M0 and M1. As a result, we could start with L3 and continue to improve our docs.

As part of the docs, it would be great to capture the rationale in the RFC as a documentation page in the docs (“docs of docs”). Where do you think such documentation fit into L3?

#23

It could easily fit into the getting started section, a “Documentation Guide” which lays out the organization and motivation, as well as how to contribute.

1 Like
#24

Thanks for your input everyone, I’m going to begin writing up L3, with the additional changes, as a formal RFC.

#25

We can continue the discussion in the RFC pull request.

#26

Pull request up: https://github.com/apache/tvm/pull/9196

Being a refactor, this is a big PR. To help evaluate it, the PR also includes a link to a rendering that you can use to evaluate the changes and more easily identify an issues.

#27

Re-spun the patch. https://github.com/apache/tvm/pull/9203