Tagging dockerhub images with "latest" tag

sanirudh · February 12, 2024, 2:58pm

Would it be possible to tag the dockerhub CI images with a latest tag and keep that up-to-date in dockerhub so that it always points to the latest tag, so that it becomes easier to track the most recent image for automation/mirroring locally within the enterprise versions.

Because of security requirements, we are not allowed to directly pull from dockerhub and hence need to maintain a local mirror of the CI images for local testing, and since “latest” tag is not maintained in tlcpack images, we need to manually mirror it locally with each image change.

leandron · February 13, 2024, 4:10pm

Hi. The issue with maintaining a latest or any other generic label that gets overwritten, is that CI would lose traceability of what version is actually being run.

You could implement some automation to know what to mirror, as the image names are in a file in the repository:

github.com

apache/tvm/blob/main/ci/jenkins/docker-images.ini

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

# This data file is read during when Jenkins runs job to determine docker images.
[jenkins]
ci_arm: tlcpack/ci-arm:20240126-070121-8ade9c30e

This file has been truncated. show original

Or maybe I missed the point of you question. If so, please let me know.

sanirudh · February 13, 2024, 4:35pm

Thanks @leandron for the reply.

I’m a relative beginner to working with docker, so please correct me if I’m mistaken here. I thought docker allows us to have multiple tags, and as far as I’ve seen, almost all other dockerhub images maintain a “latest” tag for their most recent image along with a version tag, and just version tags for the older versions. My proposal was not to change CI to use the latest tag, we can keep that part as it is (to read from the jenkins .ini file), but just to keep updating our most recent image in dockerhub to have an extra “latest” tag.

I saw that jenkins file that tracks the images. The problem is that we don’t maintain the automation of mirroring images, that is part of docker enterprise for the whole company (which by default mirrors latest tags for all the images requested), so customizing it to look at a specific file in a repo to figure out the tag is not possible.

Right now we just keep creating new requests with our docker enterprise team for each new tag update, but since there are so many CI images we maintain upstream, it gets a little too repetitive, hence the request.

leandron · February 13, 2024, 5:25pm

OK, got it. I’ve misunderstood the original ask/idea.

There is still the issue that your CI (or anything relying on latest) won’t have traceability of which version of the image it corresponds, when reproducing bugs for example. That’s one of the main reasons to use pinned image names, rather than floating tags such as latest, but I understand the complexity of aligning that with your mirroring system.

To make that something a bit more actionable, it is possible to achieve what you describe by creating some automation that will:

Read the list of images from the docker-images.ini (or check whether it changed, for example)
docker pull it locally (due to a quirk behaviour in docker, that requires the image locally for re-tagging)
docker tag it with the respective :latest e.g. ci-cpu:latest
docker push is, which should happen immediately, as all layers are cached by docker hub

Relying on people to tag each and every image with latest by the time images are updated in docker-images.ini seems error prone to me. Both our Jenkins jobs, as well as our GitHub actions have access to the token to manage images in tlcpack, maybe @tqchen can correct me here.

To clarify (as I didn’t mention that explicitly) that automation would be implemented in TVM infrastructure side: Jenkins or GitHub Actions, for example. Your mirroring job would just see :latest images being published.

As an aside, I think we were being more cost-conscious with CI, also something to keep in mind.

In case there is interest, I can expand on this suggestion.

sanirudh · February 14, 2024, 4:44am

Thanks a lot for the suggestion, that kind of an automation to tag the images would be really helpful.

I’m not sure if I would have permissions to create github actions (I have committer permissions), but I could help with creating the automation if you could provide some pointers on the process followed in general.

Thanks again for all the help.

leandron · February 14, 2024, 2:41pm

I have experience with Jenkins jobs, and a while back I contributed one to rebuild the images using TVM infrastructure:

github.com

tlc-pack/tlcpack/blob/main/jenkins/JenkinsFile-rebuild-docker-images

#!groovy
// -*- mode: groovy -*-

// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements.  See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership.  The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License.  You may obtain a copy of the License at
//
//   http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied.  See the License for the
// specific language governing permissions and limitations
// under the License.

This file has been truncated. show original

For GitHub actions, I understand how it works but don’t have a lot of experience.

sanirudh · February 14, 2024, 5:24pm

Great, I’ll see if I can create a groovy script for jenkins to tag images. I’m not sure about costs for doing this that you mentioned, but yeah let me try something like this out locally and see if I can atleast create a PR.

Thanks again