Consider refining the use of branches, tag, and automation for METbaseimage

[JohnHalleyGotway]

Describe the Enhancement

While preparing for the beta3 METplus component releases for the METplus-6.0.0-Coorindated Release, @georgemccabe and @JohnHalleyGotway brainstormed how we might restructure the branching, tagging, and automation logic for this METbaseimage repository. This issue describes the perceived problems and proposes changes to be considered and further developed.

Currently, METbaseimage contains a single branch, named main, and tagged releases are created from that single branch. The MET repository references these tagged releases to specify what base image should be used, and generally speaking, as the compilation requirements change, the METbaseimage version number is increased.

The problems with this approach include:

• The dtcenter/met-base image is only rebuilt when a new tag is created and those releases are relatively infrequent. Versions of the underlying packages can change during the intervening months. When building an image for a new release, there may be multiple problems to be fixed that originated since the last release was created.
• Once the tagged dtcenter/met-base images are created, they are never rebuilt. Instead they grow stale, and there is no mechanism for incorporating fixes for security issues.

I do note that the MET Dockerfile does run apt update to pull in OS updates that have occurred since the base image was tagged, and that is good. But its still worth addressing the stale met-base images.

I propose that we refine this approach to be more proactive about finding/fixing problems as soon as they arise. In addition, we should rebuild all active base images on a routine basis to incorporate fixes for vulnerabilities in the underlying packages. Also recommend making the branching/workflow logic for METbaseimage more consistent with the other METplus components for simplicity.

Proposed changes:

1. Rename the main branch as develop.
2. Continue creating tagged vX.Y releases from the develop branch, but only actually use those tags to document the logical changes.
3. When creating a vX.Y release, create a corresponding main_vX.Y branch.
4. Update the MET repo to replace references to dtcenter/met-base:vX.Y with dtcenter/met-base:main_vX.Y. So MET will use the contents of that dynamic main branch rather than the static tag. This insures that all downstream MET and METplus images won't contain packages that have grown too stale and contain security vulnerabilities.
5. Define automation logic via GHA to setup a cron job to rebuild all active main_vX.Y branches on some routine schedule (e.g. weekly, bi-weekly, or at least monthly). Recommend that we give some thought to what an active branch means.
6. Add new information to the METplus docs directory to define our use of METbaseimage and release process.
7. Update the MET docs directory to define where/what to update when changes are made to the METbaseimage. Currently that's defined in ~4 spots. Try to reduce that to as few as possible.

Let's discuss further and clarify these details.

Time Estimate

Estimate the amount of work required here.
Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

• Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• Select engineer(s) or no engineer required
• Select scientist(s) or no scientist required

Labels

• Select component(s)
• Select priority
• Select requestor(s)

Projects and Milestone

• Select Repository and/or Organization level Project(s) or add alert: NEED CYCLE ASSIGNMENT label
• Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
• May need a MET issue to update the dtcenter/met-base version used.
• May need a METplus issue to update the documentation.

Enhancement Checklist

See the METplus Workflow for details.

• Complete the issue definition above, including the Time Estimate and Funding Source.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
  Select: Reviewer(s) and Development issues
  Select: Repository level development cycle Project for the next official release
  Select: Milestone as the next official version
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.