Updated Docs pre-RFC

Summary

Last year an RFC was proposed to address the organization and structure of the TVM Documentation. After discussion, a first pass was made at the documentation updates following a formal documentation method developed by Divio. This pre-RFC proposes to extend the original RFC, with additions to address both user and developer audiences, as well as sub-projects within TVM.

At a high level, this pre-RFC proposes this basic high level organization structure of documents. It recognizes that while in most communities there is a distinct divide between the user and the developer communities, there can be significant overlap given the nature of TVM as an optimizing compiler.

It also addresses a common issue frequently seen with Divio style documentation, namely the addition of introduction and landing pages, as well as topic-specific pages. The next section describes the proposed high-level reorganization in detail, with two alternatives for comparison and discussion.

This pre-RFC is concerned with the organization of the documents, and not the content. As such, the implementation of this pre-RFC would move documents, and only create new documents as top-level placeholders and indexes.

The goals of this reorganization is to create a document architecture that make it clear what types of documents can be written, and how they are organized. The desired result is to make it easier for users meet their goals, whatever they are, and to make it easier for the development community to contribute to the TVM documentation.

L0 - Full User/Developer guides

This format is distinguished by putting all four document types into both the developer and the user guides. It also adds a specific section for getting started, a common feature across nearly every documentation style.

  • Getting Started
    • About TVM
    • Installing TVM
  • User Guide
    • Tutorial - A multi-chapter long-form document that to introduces users on how to install and operate TVM, with the goal of successfully introducing basic concepts and workflows.
    • How Tos - Documents that describe how to perform common tasks with TVM, divided amongst common application areas.
    • Deep Dives - Documents that cover the architecture and design of TVM, at a level appropriate for users.
    • Reference - User facing APIs, typically generated from source.
  • Developer Guide
    • Tutorial - A guided tour through the TVM code-base, using the TVMC application as a framing guide for major components of the code.
    • How Tos - Documents that describe how to contribute to particular code areas.
    • Deep Dives - Documents that cover the architecture and design of TVM, at a level appropriate for developers.
    • Reference - Developer reference documentation.
  • Topic Guides
    • MicroTVM - A collection of links to docs related to TVM
    • VTA - A collection of links of docs related to VTA

L1 - User and Developer with Pruned Subsections

The primary differences between this style and L0 are that the API reference section is broken out from both the developer and user guide into it’s own major subsection, and the deep dive section is removed from the user guide.

  • Getting started
    • About TVM
    • Install TVM
  • User Guide
    • Tutorials
    • How to
  • API Reference (flat)
  • For developers
    • Tutorials
    • How to
    • Deep Dive
  • Topics
    • microTVM

L2 - Tour style

This style takes an approach that is more focused on the user journey, from beginner to advanced with topics of interest and more focus on demos.

  • Get started
    • Installation (build from source, tlcpack.ai, etc)
    • Demo: Optimize an operator
    • Demo: Optimize a neural net (w/ TVMC, w/o TVMC)
    • Demo: Minimal deployment (w/ TVM runtime, w/ microTVM runtime)
    • Contribute
  • Tutorial
    • Relay: Import to Relay example: LeNet-5 on MNIST
    • Relay: Build Relay to binary example: LeNet-5 on MNIST
    • TE: Implement operators in Tensor Expression (compute & schedule) example: matrix multiplication
    • AutoScheduler / AutoTVM: Tune operators example: matrix multiplication
    • microTVM: deployment
    • mobile deployment
  • Architectural Guide
    • Frontend
    • Relay: Graph-level design: IR, pass, lowering
    • TensorIR: Operator-level design: IR, schedule, pass, lowering
    • TOPI: Pre-defined operators operator coverage
    • AutoScheduler / AutoTVM: Performance tuning design
    • Runtime & microTVM design
    • Customization with vendor libraries BYOC workflow
    • RPC system
    • Target system
  • Topics of interests
    • microTVM
    • VTA
  • More demos
    • Sparse models

Partial POC Rendering of L0

Guide Level Explanation

Drawing from the former RFC:

The Four Document Types

Introductory Tutorials

These are step by step guides to introduce new users to a project. A successful introductory tutorial successfully get the user engaged with the software without necessarily explaining why the software works the way it does. Those explanations can be saved for other document types; the introductory tutorial focuses on a successful first experience. These are the most important docs to turning newcomers into new users and developers. A fully end-to-end tutorial, from installing TVM and supporting ML software, to creating and training a model, to compiling to different architectures will give a new user the opportunity to use TVM in the most efficient way possible.

Tutorials need to be repeatable and reliable, because the lack of success means a user will look for other solutions.

How-to Guides

These are step by step guides on how to solve particular problems. The user can ask meaningful questions, and the documents provide answers. An examples of this type of document might be, “how do I compile an optimized model for ARM architecture?” or “how do I compile and optimize a TensorFlow model?” These documents should be open enough that a user could see how to apply it to a new use case. Practical usability is more important than completeness. The title should tell the user what problem the how-to is solving.

How are tutorials different from how-tos? A tutorial is oriented towards the new developer, and focuses on successfully introducing them to the software and community. A how-to in contrast focuses on accomplishing a specific task within the context of basic understanding. A tutorial helps to onboard, a how-to helps to accomplish a task.

Reference

Reference documentation describes how the software is configured and operated. APIs, key functions, commands, and interfaces are all candidates for reference documentation. These are the technical manuals that let users build their own interfaces and programs. They are information oriented, focused on lists and descriptions. You can assume that the audience has a grasp on how the software works and is looking for specific answers to specific questions. Ideally, the reference documentation should have the same structure as the code base and generated automatically as much as possible.

Explanations (Deep Dive)

Background material on a topic. These documents help to illuminate and understand the application environment. Why are things the way they are? What were the design decisions, what alternatives were considered, what are the RFCs describing the existing system. This includes academic papers and links to publications relevant to the software. Within these documents you can explore contradictory and conflicting position, and help the reader make sense of how and why the software was built the way it is. It’s not the place for how-to’s and technical descriptions, and instead focuses on higher level concepts.

Reference Level Explanation

This refactor will require a shift of how the documents are organized. In general, Tutorials and How-Tos are written as Sphinx Gallery documents, allowing for the generation of text, python source, and Jupyter Notebooks. This allows the user to consume these working code samples in a number of ways, but comes at the cost of fixed format that can be confusing to navigate. To help mitigate this, the tutorials and how-tos will be broken up into a more fine grained directory structure. For example:

tvm/
	gallery/
		dev_how_tos/
			compile_models/
			...
		how_tos/
		tutorial/

Rather than render the gallery in one pass as a nested structure (resulting in a single page with multiple sections), instead each directory will be rendered independently. This will aid in navigation through the galleries, and also give more fine-grained grouping of similar topics. The naming of the directory reflects the organization of Sphinx documentation folder, for example:

tvm/
	docs/
		deep_dive/
		how_tos/
			index.rst
			**compile_models/**
			...
		reference/
		**tutorial/**
		dev_deep_dive/
		dev_how_tos/
		dev_reference/
		dev_tutorial/

Depending on the type of documentation, some of the directories may be generated. For example, the tutorial and compile_models directories are auto-generated by Sphinx Gallery. To add a new Sphinx Gallery requires the following steps:

  1. Create a gallery subdirectory with the how-to or tutorial documents
  2. Create entries in the docs conf.py example_dirs and gallery_dirs variables to reference the source and target directories.
  3. Update the appropriate index pages in the docs subdirectories to add the new directories to the Sphinx table of contents.

Drawbacks

One consistent drawback of this approach is how major subprojects are handled. For example, microTVM may require a specific set of tutorials and how-tos, but these can become mixed in with other TVM specific documents. This will be mitigated through two means:

  • Subdirectories within the How-Tos can target specific topics
  • Landing pages can be created for specific topics that collect links to all of the pages related to that topic.

Another drawback is that this format may require a user to dig deeper on the first run experience, requiring them to dig into a tutorial or how-to to install the software. This can be mitigated by refactoring the landing page to include a “Quick Start” guide for installing the TVM software.

Throughout the open source ecosystem, there is often a distinction between documentation for users and documentation for developers. The TVM community is unique in that frequently users will need to extend TVM to accomplish some goal, for example adding a new backend for code generation. This issue is addressed by dividing the user and developer topics, but keeping them within the same documentation system.

Rationale and Alternatives

This style of documentation has been formalized by developed by Divio and deployed throughout the open source communities. Although it can be difficult to characterize documents within the system (“Should this be a developer or user doc?” “Is this a tutorial or a how-to?”), working within the constraints of a formalized system brings many benefits:

  • It helps prevent documentation sprawl. Rather than create new top-level headings to capture new ideas, new ideas are logically documented at different levels of detail within the for existing types.
  • It creates a consistent user experience. Users know exactly where to look depending on their needs. New users will find a path to success through tutorials, while existing users who need to solve common tasks can look to the how-tos for guidance.
  • It encourages new documentation. Developers have a framework for what docs should look like, and where they should go.
  • It takes advantage of current content. A proof-of-concept implementation of this method consisted largely of moving new documents.
  • It creates a framework to improve existing content. Many how-tos duplicate steps repeatedly. This will allow us to identify the duplications and refactor the documents into more targeted forms.

In researching documentation systems, there aren’t many formalized systems that have been published.

Prior Art

Projects That Follow This Style

Kubernetes roughly follows this style, augmented with a landing page and a getting started page.

  • Home
  • Getting Started
  • Concepts
  • Tasks
  • Tutorials
  • Reference
  • Contribute

Numpy also follows a similar style, with a very flat organization and additional documents of interest to users.

  • What is NumPy?
  • Installation
  • NumPy quickstart
  • NumPy: the absolute basics for beginners
  • NumPy fundamentals
  • Miscellaneous
  • NumPy for MATLAB users
  • Building from source
  • Using NumPy C-API
  • NumPy Tutorials
  • NumPy How Tos
  • For downstream package authors
  • F2PY Users Guide and Reference Manual
  • Glossary
  • Under-the-hood Documentation for developers
  • NumPy’s Documentation
  • Reporting bugs
  • Release Notes
  • Documentation conventions
  • NumPy license

Projects in the ML Community

PyTorch has a much more fragmented style, with Getting Started, Tutorials, and Docs (reference docs) spread across a variety of locations and using a variety of styles. The leads to a much more fragmented user experience. However, it has also been cited as a positive learning experience, and the tag search feature is powerful for the volume of documentation. Developing a similar site would likely be resource intensive.

TensorFlow follows a style that’s closer to working from beginner to advanced. One standout feature is a graphical representation of the ecosystem, with links to docs that fall into a particular categorization. When building out the developer documents, it may be worthwhile to consider a similar structure.

Other Apache Software Projects

Hadoop and Spark follow a very loose and informal documentation structure.

Sphinx Documentation Style

It’s instructive to look at the documentation style of a project for producing documentation. Sphinx follows a structure that is similar to the Divio style, but focuses more on guiding the user from getting started through advanced topics, similar to the TensorFlow style.

Non-goals

  • This documentation system only loosely addresses how subprojects should be handled.
  • It does not consider specific future documents, or a plan for refactoring duplicated content in existing documents.
  • It does not address some style issues, like how to ensure every document in a Sphinx Gallery has an appropriate image associated with it.
  • It does not address how to incorporate the new RFC process with the documentation process.
  • It does not address how to handle testing of documents and impact on CI.
  • It does not address Incorporating accepted or completed RFCs into the documentation structure.
  • It does not address the role of documentation in the CI/CD pipeline.
7 Likes

CC @jroesch @tqchen @zhiics @ziheng @yzhliu @masahi @thierry @haichen @Laurawly @merrymercy @comaniac @FrozenGene @jcf94

As we are pushing for major doc refactoring work, we would love to hear more from the community on the desirable doc structure, which is vital for our community in terms of growth and being more friendly to new contributors. Please don’t hesitate to share your thoughts! Thanks a lot!

3 Likes

I would prefer L2 in terms of more precise section titles. Sometimes it is really hard to distinguish “User Guide” and “Developer Guide”, and L2 replaces them with more explicit terms “Tutorials by example”, “Architectural guide by each component of TVM system”. The only thing missing here is API reference on the top level.

1 Like

Thanks @hogepodge for the RFC. It would be great for us as a community to converge on a documentation system that we can all collaborate together.

Based on the current proposals, I find that it might be helpful to dissect the possible choices here to facilitate the discussion(since we might want to pick some of the good choices from each of the layouts). Here are some of the choices I tried to capture from the current candidates(with possible choice labels so to help the discussion). Note this is only based on my reading and they may be incomplete(please feel free to suggest more)

Q0: Tutorial and How to/Demo Terminology

This seems to be the main difference between L1/L0 and L2

  • Q0a: Separate Tutorial from How_to
    • Tutorial is about lecture
  • Q0b: Combine Tutorial and How to
    • Rename some of the how_to to “Demo”

Q1: Whether Include Beginner Facing Tutorials in Getting Started

One pattern that L2 introduces is include beginner facing “bltiz course”(tutorial) in the Getting Started Section.

  • Q1a: Include a “bliz course” style tutorials on the Getting Started
  • Q1b: Only include landing page with links to the tutorial section

Q2: Whether we want API reference to be on top-level

  • Q2a: Keep API reference as sub section of user guide
  • Q2b: Keep API reference as a flat top level section for better accessibility.

Q3: How to Organize Developer Documentation

L0/L1 focuses on a more separated organization, by introducing tutorials/how_to/deepdive(architecture guide) sections.

L2 seems to advocate for a more component first view, by focusing on the architecture components and their relations, and embed how_tos in each of the sub component. To elaborate a bit, one possible choice could be

Architecture Guide
- Frontend
   - Description of the frontend module and its relation to the rest
   - How to add a new frontend
- Runtime system
    - description of the runtime
    - How to add a new PackedFunc
- Relay Layer
    - How to add an new operator

Notably, the way to teach the design architecture can differ from the best way to teach users. It might be more important to have a “global picture” before drilling down, this could mean that there is more weights on the architecture guide than a tutorial style guide(which I believe is what L2 tries to get at).

As a result, here are a list of possible choices

  • Q3a: Keep developer how to/tutorial, and include a L2 style architecture guide in the place of deep dive.
  • Q3b: Embed most of the how to and guides in the specific component of architecture guide, with the rationale that developers will need to first understand the component to do development.
  • Q3c: Keep an architecture guide that contains the components and their relations. Combine it with developer tutorial. Have a separate developer how to in parallel to the component guide for quick search to find actions like “How to add a new relay op”.

Q4: Choice of Specific SubTopic Page

For topics like microTVM and VTA, there are multiple ways to organize them.

  • Q4a: Organize most of the topics under the current system(e.g. tutorials/how_to) and provide a landing page in get started with links to each of them
  • Q4b: Organize the topics under a Topics section, and keep documents local to each of the subtree.
1 Like

Thanks, @hogepodge. It’s a good opportunity for us to enhance TVM documentation and tutorials together. I want to share some of my thoughts on it.

A Separated Developer Documentation

Users(who will use TVM as a tool to compile models on supported models and backends and won’t change much of the implementations) want to start quick and finally make full use of TVM API, so they need:

  1. Simple, meaningful, and runnable (important) demos and tutorials for getting started.
  2. Detailed API document to use all functions.

While developers(who will try to support new models or integrate TVM on new backends) need to know the whole architecture and design philosophy of each part, these things may be exactly what they want:

  1. A whole picture of TVM architecture.
  2. Design philosophy and terminology of each part.
  3. A few develop tutorials (e.g., adding Relay/TIR passes, adding TensorIR schedule primitives) if possible.

Nest Organization for Developer but Flat One for the Other

Nested documents are not friendly to those new users, who want to get started without knowing too much details quickly. On the other hand, developer docs may be more than user guilds. It would be great to use sub-folders to organize them.

List of Supported Ops

I think it is beneficial information for both users and developers. Users may know that why my model cannot be compiled while Developers can help to add new ops. One more step, it may encourage users become developers and enlarge the community.

One Proposal

  • Get started
    • About
    • Installation
    • Demos
  • Tutorials
  • Supported Ops (and models)
  • Python API
  • Developer Documentation
    • Architecture review
    • Frontend
    • Relay
      • Overview and Philosophy
      • Terminology and Data Structure
      • How to (add new op, add new pass)
    • TE
    • TensorIR
    • More …

Other minor points

  1. A good search engine is necessary. The current one is good but can be better if possible.
  2. The gallery is a little bit ugly without cover images.

That’s some comments and one proposal from me. Please feel free to suggest.

1 Like

IMHO, the docs should serve for two major purposes:

  1. For pure users who don’t care how TVM is implemented but just want to achieve high performance, we need to let them hands on as convenient as possible.
  2. For new developers who want to improve something with TVM, we need to let them know where to start as clear as possible.

Based on these purposes, my two cents regarding to the issues @tqchen raised:

Q1a as it has a more friendly and direct information for starters.

Q2c?: I personally don’t really care if API reference is on top-level or not. People always ignore the parts they don’t care. Instead, I think it would be more useful to separate API reference to user APIs and developer APIs. User APIs are more like user interface/programming model, such as relay.frontend.from_pytorch, relay.build, etc. Developer APIs are internal APIs such as the generated Python API reference. Note that they don’t have to be strictly separated. My point is whether pure users can find something they think useful instead of feeling frustrate when seeing tons of APIs but have no idea what’s useful to them.

Q3c: If a developer comes to the docs for a purpose, it is usually very specific and may cross multiple architecture components (i.e., “add a new Relay op”, “add a new backend”, “add a new pass”, “improve CUDA codegen”, etc). Embedding how-to tutorials to architecture components would make them lose if they have no idea what component they should work on for their purposes. Amount Q3a and Q3c, I’d prefer Q3c, because when developers find the tutorial they want, they can also learn which components are relevant and they should understand these components more.

Q4b: Given that these sub-topics are targeting to very specific audiences, decoupling them clearly to make the main page concise seems better.

Before I address the questions and options individually, I want to return to one of the motivating factors for why I feel like the proposed model will be effective. Please take the time to review this talk about the documentation system or the site describing the method as to why it’s an effective formalism for thinking about documentation.

Many of the alternatives can actually be fit into this formalism, and to me it’s useful to work within constraints and create a system that has clear expectations and boundaries. At a high level, why am I proposing this method?

  • I believe it will make for better docs, and better docs make the project better.
  • Most people want to make better docs and put a lot of work into it, but lack experience and grounding in what makes docs better.
  • With a formalism in place, it will actually make it easier to write these docs, and teach others in the community to write and contribute to docs.
  • Structure and consistency is difficult. Let’s reduce the surface area of ideas and make the structure and layout consistent.
  • Others have thought deeply about this, and just as the contributors to this project have thought deeply about deep learning and use TVM to share that knowledge and expertise, we should also learn from those who have thought deeply about documentation.

TVM has a lot of amazing documentation, capturing and distilling the expertise of the community. This is about organization, going from something like this:

to something like this:

I’m working right now on taking the feedback so far and refining the proposal based on it, and addressing your feedback. But please, if you have a moment take some time to review the source materials I’ve been working from, and that I based this proposal on.

2 Likes

Just read through this and providing my own opinions. I’m a huge fan of L2 - Tour Style here, and I appreciate that it blends topics such as TVM and microTVM in the beginning rather than treating them as separate; it makes a lot of sense to me to use this to ensure we make all of TVM approachable and as exposed as possible to potential users. As such the starting point of a quick Tutorial and How Tos for core user journeys makes a lot of sense to minimise friction. Keeping a clear divide is important here, as the core Tutorial provides the introduction whilst the How Tos provide “ok here’s your use case and what buttons to press” (I believe this is Q0a + Q1a).

As a user following this my next point of call would be to try and find a matching How To to a more exact use case, or decide to read some Explanation of that concept - which I believe the Architectural Guide does well (Q3b?).

I think this is essentially:

  • Getting Started
    • Tutorial
    • How To
  • Tutorials
    • Tutorial
    • How To
  • Specific Things
    • Topic
      • Explanation
      • How To
      • Reference

This collects Topics under the Specific Things section so you don’t have two places to look to find details on a specific topic - at the cost of one level of nesting.

I understand your sentiment here, but I would expect that divide to occur as efforts like this make TVM more approach-able and I would therefore plan for TVM being usable by a broad spectrum of users.

Following from the above, I’d question whether this is actually how we want to consider users, once the groundwork is in I would expect users to simply run tvmc or a python script against a number of developer created backends similar to how a user would use any other ML framework (without knowing the intricacies of its operation). I would agree with keeping the docs together though, the L2 - Tour Style would then let you go as deep as you need to get your model optimised.

Overall, I’m in favour of the formalisation to allow us to have discussions regarding placement and I think this provides a huge improvement in clarity.

Thanks @hogepodge for the RFC. Personally, I prefer L2 style and I think it is the easiest way to dive into TVM world. From my point of view, this style is very friendly for new users, but in the same time it is well organized, and it helps to find useful information in short time for experienced users.

Also, I think it is necessary to have well organized API reference documentation for developers. Because sometimes it is not necessary to read long documentation, but you need to find information about a function in faster way and use it. I really like how OpenCV documentation is organized. You can read tutorials about different components and modules, but at the same time you can use search to find documentation about any function and read about it. Also, for some functions, there are reference links in the description to tutorials where you can see how this function should be used.

I think, this is a good example how API documentation can be useful for users and developers at the same time. And I think that API documentation should contain not only Python part but also different internal C++ functions.

One important thing which should be included in the documentation is the current status for different frontends/backends/runtimes etc. Because sometimes we don’t know the current status of frontend/backend/runtime, and we should try to use it before understanding is it workable/usable or not. I have one real example of such situation. One my friend tried to use TVM in his own project. He used network based on YOLOv4 and used darknet frontend to convert the original model to TVM. But on the inference, he got inaccurate results. And he was able to solve this problem only by converting original darknet model to ONNX format, and then he used ONNX frontend. After these exercises, the inference gave the necessary accuracy. Also, I personally many times meet some issues in PyTorch frontend for example and only after converting the original model to ONNX format everything was fine. I suppose that it would be nice to have a good documentation about each frontend/backend/runtime etc. with specifying current status and limitations.

Reading through the comments, I want to clarify on the options presented. I had copy-and-pasted L2 from a previous suggestion, and it included more detail than L0 or L1. My concern with L2 is that it actually lacks structure, and is more similar to the current situation than a proposed improvement. Additionally it doesn’t follow the terminology as outlined in the proposal.

For example, what are demos? Are they how-tos? Tutorials? Here is expanded set of options with more details, based on existing documents reorganized into this structure.

L0/L1 Refined, expanded

  • Getting Started
    • About TVM
    • Installing TVM
  • User Guide
    • Tutorial
      • Introduction
      • An Overview of TVM and Model Optimization
      • Installing TVM
      • Compiling and Optimizing a Model with TVMC
      • Compiling and Optimizing a Model with the Python Interface (AutoTVM)
      • Working with Operators Using Tensor Expression
      • Optimizing Operators with Schedule Templates and AutoTVM
      • Optimizing Operators with Auto-scheduling
      • Cross Compilation and RPC
      • Introduction to TOPI
      • Quick Start Tutorial for Compiling Deep Learning Models
    • How To
      • Install TVM
        • Install from Source
        • Docker Images
      • Compile Deep Learning Models
      • Deploy Deep Learning Models
      • Work With Relay
      • Work with Tensor Expression and Schedules
      • Optimize Tensor Operators
      • Auto-Tune with Templates and AutoTVM
      • Use AutoScheduler for Template-Free Auto Scheduling
      • Work With microTVM
  • Topic Guides
    • MicroTVM Guide (index to existing docs)
      • → Work With microTVM
      • → microTVM Architecture
    • VTA (index to existing docs)
  • Developer Guide
    • Tutorial: Contributing to TVM
      • TVM Community Guideline
      • Performing Code Reviews
      • Committer Guide
      • Writing Document and Tutorials
      • Code Guide and Tips
      • Error Handling Guide
      • Submitting a Pull Request
      • Git Usage Tips
      • Apache TVM Release Process
    • How Tos
      • Write an operator
      • Write a backend
  • TVM Architecture Guide (deep dive)
    • Architecture Overview
    • Research Papers
    • Frontend
    • Relay: Graph-level design: IR, pass, lowering
    • TensorIR: Operator-level design: IR, schedule, pass, lowering
    • TOPI: Pre-defined operators operator coverage
    • AutoScheduler / AutoTVM: Performance tuning design
    • Runtime & microTVM design
    • Customization with vendor libraries BYOC workflow
    • RPC system
    • Target system
  • API Reference (reference)
    • Generated C++ Docs…
    • Generated Python Docs…

**L2 - to match terminology **

When you modify L2 to match the terminology introduced above, it looks more like L0/L1, but to my eyes is a bit more confusing to follow along with.

  • Get started
    • Installating TVM
    • How to: Optimize an operator
    • How to: Optimize a neural net (w/ TVMC, w/o TVMC)
    • How to: Minimal deployment (w/ TVM runtime, w/ microTVM runtime)
    • How to: Contribute
  • How Tos
    • How to use Relay to Import to Relay example: LeNet-5 on MNIST
    • How to use Relay to Build Relay to binary example: LeNet-5 on MNIST
    • TE: Implement operators in Tensor Expression (compute & schedule) example: matrix multiplication
    • AutoScheduler / AutoTVM: Tune operators example: matrix multiplication
    • microTVM: deployment
    • mobile deployment
  • Architectural Guide
    • Architecture Overview
    • Research Papers
    • Frontend
    • Relay: Graph-level design: IR, pass, lowering
    • TensorIR: Operator-level design: IR, schedule, pass, lowering
    • TOPI: Pre-defined operators operator coverage
    • AutoScheduler / AutoTVM: Performance tuning design
    • Runtime & microTVM design
    • Customization with vendor libraries BYOC workflow
    • RPC system
    • Target system
  • Topics of interests
    • microTVM
    • VTA
  • More How Tos
    • How to write Sparse models

Thanks @hogepodge for starting the discussion. In my view, I really like the idea of having the option of having a tour style organization as a way to provide a new comer a clear thread to navigate documentation. However I get how L2 could end up making the structure a little less structured at the end of the day (e.g. “having a more how tos section”). All in all, I think the proposed L1 or L0 would be perfectly adequate to organize our documentation. I’d add that it may be helpful to offer a user-journey focused thread that a new comer could use to get in an afternoon exposure to a good collection of documents.

@thierry For L2, I’m having a really hard time figuring out how to make contributions repeatable and describable. If you could expand on how the formalism behind it could work, that would be helpful to me.

Thanks @hogepodge . A minor suggestion

It would be great to leave the contributor guideline in a top-level discoverable place(how about as part of get started? or top level), since the contributor guide are something that we frequently refers to

Fair point, I agree this is the main flaw of L2. The point I wanted to make is that we can aim to have contributions be repeatable and describable on by adopting the L0 or L1 organization. A tour of TVM is possible by having a possible top level “tour page” that links to a sample of resources to make TVM approachable to a new comer. The tour is something that can require higher degrees of curation - I agree that this is something that may not be easy to self-maintain. So really that’s an orthogonal question to overall organization of documentation given that it can be prepared in many different forms.

1 Like

For a project as complex as TVM, I think the separation of User and Developer content is really important to give new people a chance at understanding what is going on and how they can leverage TVM. I think the comments on this RFC are going to be a bit biased on this point as well, since most here are already sophisticated TVM developers, and from that perspective it’s hard to remember how overwhelming all of the concepts, content, and flexibility of TVM are.

I’d lean toward L1 overall. I like the split between user guides and developer guide. Somebody who comes to TVM with an existing model to run and somebody who comes to TVM with a new piece of hardware are very different audiences, and should have different workflows.

I don’t like the split in API reference present in L0. If I’m looking at documentation and a function name doesn’t show up, my first assumption is that I’m looking at the wrong/outdate reference page, not that there was a split in the documentation.

My concern with L2 is that it while it would give the smoothest introduction to individually documented tasks, doing anything outside the explicitly documented paths becomes harder to find. Having these as how-tos, such as in @hogepodge’s expanded outline, lets there be a searchable URL, but doesn’t make them be the central feature.

Speaking from experience with the TVM documentation, one of the challenges with the tutorials that exist today is that none of them fully cover many of my use cases, so one is forced to kind of cobble together something from multiple tutorials – rather than having a clear understanding of the fundamental capabilities. This can lead to “voodoo coding” to make something work rather than getting it from first principles. Also, the different tutorials tend to have a different voice, slightly different approach to doing similar things, and there is some amount of overlap between them. It’s hard to generalize.

I think what @hogepodge is proposing for L0/L1 addresses this by getting some of the overview out of the way up front, showing the general use case with TVMC, before diving into the lower-level details. I think this is still very much about providing users an easy onboarding path, but doing so in a way that does not limit them so much. From a user perspective I think the L0/L1 approach works much better and is more in line with other OSS projects. I think it provides a better foundation for expanding the docs over time as well.

Thanks everyone for great thoughts and discussions, reading the threads, I think there are two kind of motivations that goes into this.

  • M0: The need of a Formalism in terms of separating the things into clear categories
  • M1: The need to have a tour-style map to the beginners so they can navigate

a lot of us agreed on Q0a, separation of tutorial from how to that clearly splitting the structures. On the same line, I agree that the split of developer and user facing how tos are helpful.

While L2 captures M1. We can think about is how to effectively address M1 in a document system that have L0/L1 style.

Specifically, we might be able to tweak the Getting started section a bit to incorporate some of the elements guided map elements(e.g. like Q1a) but keeps the rest of the section as clear L0/L1 style separation.

BTW, might be not strictly related to this thread, but could be good to bring it up: shall we have a workflow figure like this the one blow + plus some missing layers? (introduced in this RFC)?

image

We may link each component in the figure to their corresponding docs. For example, the logos on the first row link to importer docs; the second row to Relay, etc; The optimization column can be replaced with specific topics: VTA, microTVM, AutoScheduler, etc

1 Like

I believe this would be a good navigation page for the architecture guide, and can fit into both L0/L1 and L2 style