Recent CI Improvements

[driazati]

I, along with @areusch and others, have been working on improving TVM’s CI and developer experience recently after the launch of our new Jenkins instance earlier this year. This is a quick summary of some of the highlights:

CI Runtime

• #10359 Use sccache and all available CPUs in builds
  • Should improve build times by 40x from 40 minutes to 1 minute for Python-only changes and PR re-runs
• #10414 / #10413 - Shard frontend: GPU job into 2 jobs
  • Add job level parallelism in Jenkins for the slowest CI job, frontend: GPU splitting a 3 hour run into 2 concurrent 1.5 hour runs
• #10247 Update CI to Python 3.7 and Ubuntu 18
  • Use up-to-date versions of Python and Ubuntu and soft-remove support Python 3.6 and Ubuntu 16 (these are no longer tested)
• #10121 Fix JUnit failure reporting
  • Enable Jenkins’ test report viewer so developers don’t need to trawl through logs for Python test failures, e.g. https://ci.tlcpack.ai/blue/organizations/jenkins/tvm/detail/main/2635/tests

CI Reproducability

• #9534, #9971, #10425 (ongoing) - Add tests/scripts/ci.py to replicate CI tests locally
  • This script runs in the same Docker images and in the same environment as CI, making it easier to replicate CI failures locally.

Workflow

• #10057 Add @slow decorator to run tests on main

  • Add a decorator @tvm.testing.utils.slow that can be used to run marked tests only when a CI run is invoked from a commit on main, and not on PRs. This can be used to add more long-running tests to CI without fear of slowing down the CI critical path.

• #10056 Add auto-updating last-successful branch

  • This adds a branch last-successful to TVM that tracks the last main commit that passed CI. This can be useful to ensure you are basing your work on a green commit.

• Marking tests flaky

  • We have been more active in labelling flaky tests and putting up associated PRs to disable these tests. Please don’t hesitate to file an issue if you suspect a test failure in CI is not due to the changes being tested.

• RFC to replace CODEOWNERS with automation

• RFC to enable better commit messages and make merges easier