@leandron thanks for posting this! including some of my thoughts replying to your questions:
Q1: Where do we upload the staging images to?
this seems reasonable. Probably a new DockerHub organization called that, and the bot can be a DockerHub account called tlcpack-ci-build-bot.
Q2: what notification we send when a job fails?
either an e-mail list or a discord notification seems good. @tqchen can comment whether discord is sufficient.
Q3: How to tackle issues coming from the Docker images rebuild?
We will need to do this as a community. I think Discord may be a good place for these higher-bandwidth debugging conversations to start. however, we should establish a process for logging failures so that we can keep track of them. in my experience, there is often a long tail of difficult-to-trigger, flaky CI failures. when you have these, you want to create a set of e.g. GH issues and link to the Jenkins log for each of these runs. After you’ve accumulated enough logs, you’ll begin to form theories; at this point, the list of links to failure logs is invaluable to resolving the underlying issue.
I agree discussing this further at community meeting makes sense to me. There are some open questions as to whether we have an “on-call” or something.