Request for Comment: Possible migration to Bootable Containers

[JohnMertz]

I've made a comment in this issue with this idea, but will try to be a little more clear with the thoughts behind this are here and then welcome all comments (TL;DR at the bottom, but please read everything if you would like to comment).

Note that I'm not unbiased. I've been thinking about this for a few weeks and am very enthusiastic about the idea. I will try to be clear with the advantages and also the draw-backs, but note that I will be using a lot of negative language to describe the current distribution model since I have vivid knowledge of the problems involved and want to make it clear what problems I'm looking to solve.

The Problem(s)

There are several challenges with distributing software in the way that MailCleaner has been (ie. long-running VM images with the software spread across the filesystem and no mechanism for update integrity):

• Builds are not reproducible - MailCleaner was originally built from a generic Debian image with the the software installed on top. Since then, each new image is simply an updated version of that original image. It would be impossible to go back to the first build, pull down all available updates and end up with an identical image.
• Almost all installations are different - Depending on when a machine was installed, it will have a different starting point. From that point forward, different machines may have succeeded or failed to install different updates which can then cascade into other failed updates, etc. This requires extra caution to have the application accommodate for a lack of updates (eg. by checking the version of specific packages before writing config files).
• It is impossible to simply rollback from a bad update - Major updates to MailCleaner are applied via "Updater4MC" scripts. These apply changes directly to the root operating system. If there were unforeseen differences on a specific installation or a bug in one of these updates which leads to problems, then it is necessary to create new updates to attempt to reverse the changes. Sometimes reversing changes is impossible if there is data loss, corruption, or if the number of possible side-effects is too numerous to distill into a common update.
• OS upgrades are very difficult - As demonstrated by the fact that MailCleaner is still using Debian Jessie (and some stubborn users who were still using versions older than that the last time that this data was available), it is very difficult to upgrade to the base OS. Enough changes build up between releases which makes it unreliable to have the same application code running on two different OS version, so a different build is necessary. For MailCleaner, this means a different GitHub repository and updater. There are also enough custom MailCleaner changes layered on top of Debian to makes it impossible/dangerous to simply migrate OS versions with a standard apt-get dist-upgrade even if the application code could easily be migrated. This means that it requires a migration to a new VM.
• Builds are inflexible - MailCleaner images get cleaned and converted to each of the other VM formats (OVA, VHD and VHDX from QCOW2). However, it doesn't have the flexibility to deploy to any other formats (eg. installable ISOs, OCI images, alternative architectures etc.). Some of the build process is automated, but typically the existing image packer requires starting from a QCOW2 image which has been fully updated manually.
• Hosting VM images is expensive or challenging - The community download page doesn't even exist anymore. Each of the 4 VM image is just over 1GB and can be downloaded hundreds or thousands of times per month. MailCleaner used AWS until it got too expensive and then used a dedicated Alinto webserver. As a community run project, we would need to find affordable hosting, which would likely mean limited formats and only one release available at a time.

The Proposal

Technology to produce bootable container images is rapidly evolving. The foremost of these seems to be the BootC project, lead by the Cloud Native Computing Foundation (CNCF) the stewards of many other well-regarded projects such as Kubernetes, Podman, Prometheus Helm and many others.

The proposal would be to migrate to deploying via BootC OCI containers. These provide a read-only root filesystem which is versioned and atomically updated so that all installations will have an identical root filesystem and updates are guaranteed to install successfully, including across OS versions. The "bootable" component means that these containers can be installed to bare metal or a hypervisor by providing a bootloader and kernel so that it can operate as a traditional server/desktop without a container runtime and host OS (although it would remain possible to run under a Docker/Podman runtime also). Open Source tools exist to automatically build and host images and ISOs.

This would solve all of the above problems:

• Builds will be reproducible - As I work towards the first release of SpamTagger Plus, I will have a fresh start where I set up builds to install programmatically in a fully Open Source way. So this may be solvable without BootC. With BootC however, building in this way is required (ensuring that discipline around this workflow is maintained) and would be be able to be replicated identically on any user/contributor's local machine without requiring a pre-existing VM image.
• All installations will be identical - The root filesystem is installed from the OCI image and is read-only. When the system updates, a new version of the image is installed separately and will be booted from the next time that the container/VM reboots. The SpamTagger Plus application would be deployed inside the read-only image rather than being downloaded directly from GitHub. It's version would be guaranteed to match with the remaining OS files so it would be impossible for it to be missing a necessary OS update.
• Rollback is simple - The previous BootC image remains installed and even earlier version are installable for as long as they remain on the registry. In the event that a new release was not well tested and causes problems, you can return to the previous state by running bootc rollback and reboot to return to the version you were running prior to the last update.
• Distribution upgrades are relatively trivial - We would still need to accommodate for major package changes, but generating the base image would be automatic and updates would generally be limited to the applicaction code only. When the new distribution version is available, it can be installed identically to any other update (ie. download a new OCI container and reboot).
• Builds would be more flexible - All BootC images start from a generic OS image and then have all additional packages and configurations applied programmatically during the build. This allows for different tagged versions to be built (eg. 'stable' and 'testing'), and for the the final OCI image to be built for different architectures. Those images can then be made into installable ISOs, converted to all VM image formats, and can be pushed to cloud provider marketplaces automatically using open source tools. The SpamTagger application could be added to the images by simply copying files into filesystem during the build, so it does not even require knowing know how to create an RPM package (although it would be nice to deploy as a package to COPR as a non-official installation method).
• Images can be hosted for free on GitHub and other container registries - GitHub allows for a limited number of "Packages" to be hosted for free and significantly more to be hosted at relatively affordable rates. Alternatively, any other container image registry can be used to host the images, including self-hosted options or other community registries.

Sounds good? What's the catch?

These improvements are huge! However, nothing is without it's downsides.

• Debian is currently not supported by BootC - BootC currently has some hard dependencies on the RPM ecosystem. There are currently projects using BootC for CentOS, Fedora, and other derivatives. Support for Debian is a work is in progress. However, unless that sees significant progress in a relatively short perioud of time, we would probably need to migrate to CentOS.
• Reboot is required to install updates - Since the root filesystem is read-only and updated all at once, a reboot is required to start using the new filesystem image. A reboot is required even if only a couple of files have changed which would not require a reboot on a traditional system. SpamTagger Plus will be a fairly slow developing project after the first major release is complete, so this should not require overly frequent reboots. A systemd unit would be created to detect when a reboot is necessary and would automatically reboot according to your user preference (ie. specific times and days of your choosing) with an alert email before doing so.
• Applying custom configurations is more difficult - Again, since the root filesystem is read-only, you can only apply changes to /etc and /var which are writable. Custom templates and other supported customizations would be fetched from /etc/spamtagger instead of the current location in /usr. Additional software must be installed via Docker/Podman, Brew, PIP, NPM, Distrobox, or any other package manager which can install to a writable directory. Some documentation would be included on this, but unsupported modifications would be discouraged. We would prefer that additional packages just be added to the official builds after some testing if they may be applicable for other users. We could potentially have different tagged releases for add-ons (eg. 'stable-eset' to add ESET antivirus to the stable release) which would come bundled with the read-only image and you would add or remove these by using bootc switch to reboot into the version with/without that package.
• This is unfamiliar and fast-evolving technology - Users may be uncomfortable using a different operating system, different paradigm for installing additional software and different update process. The tools surrounding BootC, including build scripts are likely to change a lot so it may require some additional work outside of the application code to maintain and improve processes. However, the current state is miles better than what we have with MailCleaner and each change is likely to be an improvement.

My experience

I've been using similar technology on my desktop computers for years, starting with Fedora Silverblue 32 and Fedora CoreOS for my server since version 38. These use the same paradigm for distributing the operating system, whether you call it "immutable" (the common description, which advocates are trying to move away from), "atomic", "cloud native" or "image based". From these and other Fedora Atomic desktop spins, I've moved on to UBlue which is the foundation of the very popular gaming OS Bazzite which demonstrates feasibility for an appliance-like device (although I'm not a gamer, so I've never personally used it).

I've been getting some experience with these tools from a development perspective over the last couple of months. UBlue has recently moved to using BootC to provide bootable container images instead of using the older rpm-ostree tool to install and manage the read-only filesystem snapshots and has dropped support for my preferred flavour (the Sway window manager version), so I've taken to building my own BootC images for my desktops.

TL;DR

The proposal is to change SpamTagger distribution to use bootable containers instead of perpetual VMs:

	Current (MailCleaner)	Proposal
Base OS	Debian	CentOS
Installation	Installation from a small selection of x86_64 VM image formats.	Potentially install from a wider range of VMs, ISOs, containers, cloud provider marketplaces (ie. AWS, etc.) and other methods and architectures.
General operating principle	Functions as a general-purpose operating system with the email filtering application pre-installed	Functions as an email filtering appliance with general-purpose functions discouraged and sandboxed
Application updates	Applied by pulling from GitHub directly and with install scripts which must attempt to accommodate for an indefinite number of configurations and have no good mechanism to verify integrity before or after updates. However, it only requires restarting services after the update; no reboot of system required.	Download new, pristine images with each update and guarantee that the update installed successfully after reboot (ie. "atomic" updates)
OS upgrades	Impossible to do safely on the same installation. Requires new installation and migration of data.	Functions like a normal update. The new image you download will include all of the new OS packages and is again guaranteed to install successfully and be usable after a reboot.
Filesystem access	Fully writable filesystem which can accumulate idiosyncrasies and incompatibilities over time.	Only /etc/ and /var are writable. The base operating system and application code cannot be changed except by updating and rebooting to the next pristine image.
Additional applications and configurations	You have full access to apt-get and dpkg as well as the ability to apply changes directly to the root filesystem and application source code. However, modifications and adding additional packages is - and has always been - strongly discouraged.	Additional applications and configurations can be added to /etc and /var without risk to the operating system or application. This means that you can install via methods like Docker/Podman, Brew, PIP, NPM, Distrobox, etc.
Bugs and support	The system can easily accumulate non-standard configurations and failed updates which can continue to compound until the system is unrecoverable and requires a fresh install. User actions can easily break the system. Niche bugs will continue to impact only one or a small number of users. We no longer have Enterprise support mechanisms in order to access and investigate these issues.	All systems are guaranteed to be identical (other than the configuration of the application) and thus all bugs are reproducible on any machine with minimal details on how the problem machine is configured.

Feedback

What are your thoughts on this possible change?

I don't expect anyone to be an expert on any of these tools and methodologies in order to comment. I'm mostly looking for general sentiments.

Package availability on CentOS Stream 10 are not significantly different from Debian Bookworm or Trixie, so any work that still needs to be done for a new Debian release still needs to be done for CentOS and not a lot of significant new work would be required. However, it would still be better to begin a transition sooner rather than later. The most significant difference would be in the build infrastructure, but given that there are already other projects being built using CentOS BootC images it will probably actually be easier to proceed under the proposed scenario since much of that work is already being done for us by another active and excited team of developers.

[MHKnowles]

Hi John, Mike here again.

I would be all for a portable containerised SpamTagger IF there were bulletproof "how to" guides on how to spin up the latest Linux/whatever build ground-up from the OS to include adding a containerisation system, and everything else needed to get it working to the extent that I can use a GUI to configure it. The reason for the GUI is because some of us have little or no Linux/Unix knowledge (as you know) having come from a Windows background, and if you want to appeal to those of us running MS Exchange deployments (which is likely to be a large proportion of your userbase) then we're used to the Windows way of doing things.

[JohnMertz]

Hi Mike,

I did reply to you via email regarding SpamTagger (Plus) maintaining a GUI (regarding certificate generation) but I'll paraphrase for everyone else:

Yes, you will always have a GUI for the MailCleaner continuation (currently "SpamTagger Plus"). Almost all features have always been introduced via the commandline first, because it is much simpler, but everything is eventually given a GUI method of configuration if appropriate.

The future project, currently at the placeholder SpamTagger repository, pending renaming will eliminate a native Web UI in favour of an API to integrate into other products or custom UIs. However, this will be a separate product and it will not see a release for at least a couple of years at this point. The direct MailCleaner successor should look and behave largely identically to how it does now, in perpetuity.

Regarding installation: The primary installation method will be almost identical to MailCleaner. ie:

• Download the appropriate VM image for your environment: QCOW2, OVA, VHDX (potentially adding AMI and other formats for directly launching a VM to AWS or other cloud hosts).
• Load it into your hypervisor
• Upon first boot you'll have to log in with the default username (spamtagger) and password (STPassw0rd) and it will then automatically run the commandline installer to: change keyboard layout, timezone, root password, and configure networking

The main difference will be that MailCleaner has a second-stage install where, after the above, the "configurator" interface should become available via the web on port 4242. There, it will ask for you to set a host ID, create a web admin password, and configure the base URL. To me, this has always been a needless diversion since the prior commandline installer where you just set the root password, etc. is equally capable of configuring these settings. The configurator also requires you to manually navigate to correct hostname/port (often requires manually overriding the DNS resolution for the 'mailcleaner'). It can also open your system up to being easily compromised if you leave the configurator open to the public (since it can be used to reset the admin and root passwords.

Instead I've just added a "SpamTagger Plus configuration" set to the end of the exact initial commandline wizard, since you have to do all of the prior steps there anyways. This asks for a all of the same details that the configurator already did without needing to navigate anywhere else. It does not require you to know any additional Linux commands, etc.

This all has limited relevance to the Bootc migration.

The primary difference with Bootc-based images will be that the root filesystem is read-only, so you won't be able to modify any files other than those in /etc/ (where your custom configurations and templates will be located) and /var/ (where the bulk data will continue to be located, ie. quarantines, logs, virus signatures, MTA queues, etc.). The tools needed to access and manipulate these files will be identical to MailCleaner, only a handful of file locations will change.

Regardless of what the installation method is, day-to-day configuration will still be via the Web UI.

There will hopefully be a number of supported installation options, aside from the official VM images, although I expect a very small number of people to use these alternative options. They will mostly exist because I want SpamTagger to be accessible to users in any environment and to provide easier ways for contributors to get write access to the filesystem in order to apply and test changes. I'll document all of these thoroughly before recommending that anyone try them.