[DISCUSS] Upgrade overall CI dependency

Right now CI runs on ubuntu 18.04, with python 3.7 support. We have used this config for a while. A few of things are getting out of dated this year,

  • Ubuntu 18.04 LTS supports ends April 20 2023
  • Python 3.7 supports ends on Jun 2023

Similarly, cuda now comes with CUDA 12, and most of the frameworks are at least cuda 11 now. PyTorch 2.0 now requires python 3.8 support.

There can also be possible libstdc++ compact issue for later versions of LLVM build. So it would be a good time for us to discuss an update of the CI image. Some discussion items to consider.

  • Which Ubuntu image to update to (we can choose 22.04 if we want a longer term support, or 20.04)
  • What would be the python dependency requirement, would updating to 3.8 makes sense given the related package ecosystem moving that way
  • Options to enable cuda versions, likely latest cuda 11 and optionally cuda12 could be a starting pt.
1 Like

I use TVM on a variety of environments, including some development boards that are running much older versions of Ubuntu (there have been no updates from the vendor).

To make my life easier, I use Docker for all of my workflows, and that means that TVM doesn’t need to worry about the version of Ubuntu or python running on the host, as long as the image has the right version. For that reason, I think that it makes sense to move these components to the longest term support versions, e.g. 22.04.

However, on devices where I have CUDA, I believe that the host CUDA version is important, regardless of what is happening in an image. Therefore, although I think it’s important to test newer versions of CUDA, and ideally exploit any new features that it has, we should keep older versions of CUDA in CI for as long as possible, as losing support for a CUDA version is harder to deal with since end-users are forced to update their hosts, rather than simply update a Docker image. In many scenarios that may be more difficult or impossible.

One thing I found really needs upgrading: pylint has been quite outdated and its pylintrc has become incompatible with mine locally. It might take some effort to fix pylint though if we want to upgrade it.

Regarding CUDA: CUDA 11.x really spans from 2020 to 2023, and the performance/functionality change a lot throughout its release. We might want to keep ourselves up-to-date.

Regarding LLVM: we might want to consider the latest LLVM + (latest LLVM minus 1) to ensure we are compatible timely.

A further comment on python: Right now we are using black as formatter, but the imports are not sorted yet, which may hide some cyclic dependency issues. I would encourage that we introduce isort as an extra formatter during the upgrade. Alternatively, latest python tools like ruff might be useful in our case.

I’m happy to chime in to help!

5 Likes

Now that we’re producing stable builds, can we carry an up to date version of LLVM and provide it to any users bundled with the package? The packages should be built using manylinux so will be compatible with most modern systems.

It’d be really good to get dependabot or similar to help us in rolling updates of this so the next update is more incremental? If we ever find out one of our dependencies has a security vulnerability we may find it equally painful to upgrade.

I would be very excited to try a tool such as ruff to consolidate our tooling :heart_eyes_cat:

3 Likes

This is a great idea! Ideally upgrade should be as incremental as possible so breaking doesn’t happen that often. We do have a dependabot right now, but unfortunately those PRs from the bot don’t usually get enough attention :frowning:

Yep and it should suffice in most of the cases. Meanwhile, there are needs to integrate with nightly LLVM to use their latest features, so I was thinking if we could have a nightly CI to test it out

I also support moving to ubuntu 22 to have longer support. Ubuntu 20 means we have to update dependencies in less than two years from now. I think Ubuntu 22 could be default choice for our images, unless if any specific target is not compatible.

As we go through this major change, I also suggest to develop a base docker image with common dependencies and build the other images based on this base image. This would remove the duplications that we have across multiple docker files.

I would normally err on the side of caution and say we should stick with the oldest supported version of each dependency in CI for stability. For Python I think this matters so we should move to Python 3.8 and generally stay towards the tail end of support. But for Ubuntu I haven’t seen personally cases where there are bad compatibility issues so I support moving that to 22.04 to make CI easier to live with. As for CUDA it might be good to run a small smoke test with CUDA 12 but run most of the tests on a recent CUDA 11.

The needs for release builds are totally different though and these decisions shouldn’t really affect them, those could be offered with a variety of configurations of easy installation (which we do today with multiple CUDA versions) but the matrix can blow up pretty quickly so as a general policy the dependencies we release should be on the latest stable release.

Specifically on the question in OP.

I suggest we move to 22.04, which is already 1 year old, and then we install Python versions we need from something like deadsnakes repo.

Yes, I agree with 3.8. Perhaps packaging needs to be expanded for 3.11 (I’ll have a look).

No strong opinion here as I don’t know much, but whatever it is, I think it would be cool to make sure packaging is covered. Maybe even deprecating the 10.x versions we have, and focusing on the new ones?

Is this only for the CI images, or are you suggesting that we change the minimum OS, Python, and LLVM version requirements for ongoing development?

it is only for CI images. There might be some implication on minimum versions which we can also discuss. For example, I think moving to python3.8 when 3.7 is EOL seems to be a good move(following PT)

Thanks everyone for chiming here. Let me propose a strawman based on the inputs so far:

  • Ubuntu docker: move to 22.04
  • Python move to 3.8, with consideration of deprecating 3.7
  • Cuda, follow PT, and support cuda 11.7 and cuda 11.8
  • LLVM: move to latest LLVM (perhaps minus 1 or two)

Linter:

  • Update to latest version of pylint, we can discuss new linter alternatives like ruff as well, as a separate thread
1 Like

Seems we have a good amount of agreement here, leaving this thread open for another three days, then we can proceed with the plan

2 Likes

I want to update to Python 3.8 to unblock PT 2.0 on CI. Can I start working on it?

1 Like

That sounds perfect!

As proposed by TQ. The CI dependency upgrade has been completed.