[RFC] TVM Documentation Refactor

[hogepodge]

Introduction

A major component of building a community is making it easy for new contributors to become successful with the project, while giving clear reference materials for experienced contributors to build upon and use existing work.

A strong documentation system should support the entire lifecycle of a community member. This RFC proposes a refactoring of TVM documentation to better support the entire community. Based on the system developed by Divio, this refactoring will focus on creating four major categories of documentation:

• Introductory Tutorials: Practical documents that are most helpful in helping new users learn the software.
• How-to Guides: Practical documents that are most helpful in getting work done.
• References: Advanced documents that are most helpful in getting work done
• Deep Dives: Advanced documents that are most helpful in learning they why’s of the software.

The Four Document Types

Introductory Tutorials

These are step by step guides to introduce new users to a project. A successful introductory tutorial successfully get the user engaged with the software without necessarily explaining why the software works the way it does. Those explanations can be saved for other document types; the introductory tutorial focuses on a successful first experience. These are the most important docs to turning newcomers into new users and developers. A fully end-to-end tutorial, from installing TVM and supporting ML software, to creating and training a model, to compiling to different architectures will give a new user the opportunity to use TVM in the most efficient way possible.

Tutorials need to be repeatable and reliable, because the lack of success means a user will look for other solutions.

How-to Guides

These are step by step guides on how to solve particular problems. The user can ask meaningful questions, and the documents provide answers. An examples of this type of document might be, “how do I compile an optimized model for ARM architecture?” or “how do I compile and optimize a TensorFlow model?” These documents should be open enough that a user could see how to apply it to a new use case. Practical usability is more important than completeness. The title should tell the user what problem the how-to is solving.

How are tutorials different from how-tos? A tutorial is oriented towards the new developer, and focuses on successfully introducing them to the software and community. A how-to in contrast focuses on accomplishing a specific task within the context of basic understanding. A tutorial helps to onboard, a how-to helps to accomplish a task.

Reference

Reference documentation describes how the software is configured and operated. APIs, key functions, commands, and interfaces are all candidates for reference documentation. These are the technical manuals that let users build their own interfaces and programs. They are information oriented, focused on lists and descriptions. You can assume that the audience has a grasp on how the software works and is looking for specific answers to specific questions. Ideally, the reference documentation should have the same structure as the code base and generated automatically as much as possible.

Explanations (Deep Dive)

Background material on a topic. These documents help to illuminate and understand the application environment. Why are things the way they are? What were the design decisions, what alternatives were considered, what are the RFCs describing the existing system. This includes academic papers and links to publications relevant to the software. Within these documents you can explore contradictory and conflicting position, and help the reader make sense of how and why the software was built the way it is. It’s not the place for how-to’s and technical descriptions, and instead focuses on higher level concepts.

Proposal

Following the model outlined above, the RFC proposed the following high-level structure for the TVM documentation with specific suggestions for individual documents.

• Landing page
  • Introduction to TVM
  • Links to major sections
  • Full table of contents
• Introductory Tutorial
  • Installing TVM, import a model, and compiling it to target platforms
• How to
  • Installing TVM
  • Contributing to TVM
  • How-tos that are described as tutorials in the current documentation
• References
  • Language reference
  • Current API reference
  • Specifications(e.g. log format)
• Deep Dive
  • Design and Architecture
  • Literature
• VTA
• FAQs

Phases

The documentation refactoring will follow the following phases:

P0: Preliminary work

Preliminary work to reorganize the documentation structure. Some of this was completed in PR #6146.

P1: Categorization of existing documents

Creation of new documentation categories in existing code base, along with top-level landing page.

P2: Restructuring of existing documents

Existing document will be refactored into the new categorization framework.

P2+: New documents

New documents will be contributed by the community according to this structure, in line with the sections outlined above.

P3: End to end introductory tutorial

A full end to end tutorial will be written that describes how to install TVM and supporting software, importing a well-known trained model, and compiling it to different target architectures with optimizations.

[tqchen]

Thanks @hogepodge for the proposal, with some prelimary effort along the lines of P0 and P1, we have already make the structure to roughly reflect the overal organization https://tvm.apache.org/docs/

[hogepodge]

Following up on this, I’m preparing a pull request that will move existing documents into the new structure. Aside from new structure, it won’t add new content, just move existing content. A copy of my working sheet is here. There are a few things to note:

I’m making a distinction between user documentation and developer documentation. This is a common practice and helps make it clear what aspects of TVM are user facing and which involve direct contribution to the project. The nature of TVM causes this line to be blurred a bit, but as we grow the community it’s an important distinction to make.

Many of our current “tutorials” are actually “how-tos”, and a good amount of the work is in moving thest to the correct location.

I’ll be working with @areusch to determine the best location for micro TVM documentation.

I wanted to send this up for comment before I got too deep into the pull request. Because it’s an organizational refactoring, a lot of files will be touched. Again, not new content, but moving of existing content.

Thanks, Chris

[areusch]

IMO as the project develops and sub-projects or topics emerge, we need a top-level landing page to organize the tutorials/docs related to those sub-project/topics. I’ve organized the microTVM docs around that concept–the landing page is here. A similar example is VTA. I think we could also have similar landing pages for topics like Quantization, Autotuning, Integrating TVM with hardware, etc. I’d suggest we create a section for these larger landing pages, such as “Get started with …”

[hogepodge]

To clarify my previous post, the TOC tree would look something like this.

• TVM User Guide
  • Introductory Tutorials
  • How To Guides
  • Deep Dive
  • Reference
• TVM Developer Guide
  • Getting Involved with TVM (tutorial)
    • A Tour of the TVM Code Base
  • How To Guides (domain specific)
  • Source Deep Dive
  • Reference

Addressing @areusch’s sub-project landing pages, I’m not 100% certain how to incorporate that, as each sub-section (tutorials, how-tos, and so on) would carry pages for each topic. One idea would be to create specific landing pages that filter to specific document types. Another would be to break up into more top-level nodes, which isn’t entirely satisfactory to me.

I’m also open to working through a completely different organization.

[areusch]

@hogepodge i agree it doesn’t make sense given our current tutorials. But i think many of the tutorials need to be deleted and rewritten. for example, consider “Compile Deep Learning Models” doctree:

Compile Deep Learning Models

• Compile PyTorch Models - teaches how to import from PyTorch then build the model, run it on CPU, then lookup the classification output in a table.
• Compile Tensorflow Models - ingest a model into TensorFlow, use PIL to decode an image, import the TF model into TVM, then build and run either on CPU or GPU, then lookup classification output in a table
• Compile MXNet Models - download resnet18, import into TVM, compile using CUDA, execute on GPU, then lookup classification output in a table. finally explains how to use aux params.
• Compile ONNX Models - download an image scaling model, import into TVM, compile, run on CPU, format the output using PIL/pyplot
• Compile Keras Models - download Resnet50, import into TVM, compile using CUDA, execute on GPU, then lookup classification output in a table.
• Compile TFLite Models - download mobilenet, import into tvm, load an image with PIL, compile to CPU, run inference, lookup classification output in a table
• …

80% of each of those tutorials is repeated. Meanwhile, they don’t even start to talk about how to get the dependencies installed properly. The TensorFlow tutorial references (without linking to) some non-existent page docs/frontend/tensorflow.md which ostensibly should have explained this. These tutorials are grouped under “compile deep learning models” but most of their content is demonstrating how to get the end of model inference and show some result of running the model rather than actually explaining various caveats of compiling it.

The tutorials are not really helping past basic comprehension here. They could be much smaller or more targeted:

• Just show how to install each framework’s dependencies so TVM can use it, and how to get a Relay model
• show how to compile a relay model for various platforms
• show how to run a variety of different models (e.g. mobilenet, resnet50, etc). we could probably just show some of these on CPU to make it possible to run the tutorials without needing a GPU.

if we do this split, then having the landing pages makes way more sense, because they sew together each of these smaller tutorials. i do still think it makes sense to have some end-to-end examples, and even the intro tutorial you put together could be a good one of these. I don’t think we can continue to have the cross-product of all possible models from all possible frameworks to all possible targets. I would like us to refactor the tutorial and then write higher-level documentation to explain the process (with pictures) at a high level in terms of core compiler data structures (e.g. Relay IRModule, runtime.Module, etc) and then provide a tutorial to give detailed steps for each logical step in the process. Doing this makes it easier to leap from basic understanding of tutorial to basic understanding of compiler structure.

[zackcquic]

Thanks a lot @hogepodge

I am wondering, in your plan, would there be model zoo pages describing current support models for each frontend along with runnable/testable scripts?

[hogepodge]

It’s not in my backlog for this quarter, but it’s something we could look in to.

[hogepodge]

I’ve attached a couple of screenshots, and a PR of an incomplete POC of the refactor. [WIP][POC] Proof of concept for documentation refactor. by hogepodge · Pull Request #8676 · apache/tvm · GitHub