Signed, immutable bootc images with pre-compiled ZFS storage and GPU kernel modules. Built entirely from upstream vendor packages, automated via GitLab CI/CD with full supply chain security.
- CentOS Stream 10 variants: ZFS storage, NVIDIA GPU
- Fedora 43 variants: Minimal base (modern kernel)
This project gets kernel modules and drivers directly from their upstream vendors, not from intermediary packagers like rpmfusion:
- ZFS from the OpenZFS project's official repository
- NVIDIA drivers from NVIDIA's CUDA repository
Intermediary packagers are often volunteer-maintained and can stall, lag behind, or break without warning. Upstream vendors have market pressure to maintain their distributions. By depending on upstream directly, the project avoids single-maintainer risk in its supply chain.
Kernel modules are compiled at build time using DKMS in isolated builder stages. The final images contain only the compiled .ko files and userspace binaries - no DKMS, no kernel-devel, no gcc, no compilation toolchain. This multi-stage separation is the core architectural invariant of the project.
Every image published from the main branch is cryptographically signed using Sigstore cosign via GitLab's OIDC identity. CycloneDX SBOMs are generated and attested alongside each image. There are no long-lived signing keys to compromise - signing is keyless and logged in a public transparency log (Rekor).
Storage (base-zfs, base-zfs-nvidia):
- ZFS 2.2.x - Kernel modules + userspace tools (zpool, zfs commands)
- Data pools only (not root filesystem)
GPU (base-zfs-nvidia):
- NVIDIA Open Kernel Modules - RTX 20/30/40/50 series, GTX 16 series, data center GPUs
- NVIDIA Container Toolkit - GPU access for rootless containers via CDI
Minimal base (fedora-base):
- Modern kernel with latest hardware support
- Same system utilities as CentOS base
- Ideal for containerized GPU workloads (ROCm containers, vLLM, etc.)
AMD GPU kernel module builds (amdgpu-dkms) are disabled due to kernel incompatibility. The driver fails to compile against kernel 6.12+ (missing dma_resv->seq API). See ROCm/ROCm#5111 for upstream tracking.
Recommended alternative: Use containerized ROCm with the in-kernel amdgpu driver:
podman run --device=/dev/kfd --device=/dev/dri \
rocm/vllm-dev:rocm7.2_navi_ubuntu24.04_py3.12_pytorch_2.9_vllm_0.14.0rc0 \
vllm serve <model>The in-kernel driver provides /dev/kfd and /dev/dri for compute workloads without needing amdgpu-dkms. See AMD ROCm documentation for container setup guides.
- Monitoring: vim, btop, lm_sensors
- Networking: mtr, iperf3
- SELinux: setroubleshoot-server
- Utilities: tree
Tier 1: Upstream bootc (mirrored + signed)
├─ quay.io/centos-bootc:stream10
│ └─ Tier 2: base → base-zfs → base-zfs-nvidia
│ └─ Tier 3: your-instance (FROM base-zfs-nvidia + users, secrets)
└─ quay.io/fedora-bootc:43
└─ Tier 2: fedora-base
└─ Tier 3: your-instance
Tier 1 = kernel, systemd, core OS. Tier 2 = compiled kernel modules, utilities, signature policy. Tier 3 = machine-specific config (users, SSH keys, services). Instance repos stay separate for secret isolation and independent CI/CD.
DKMS compiles modules at build time. Final images contain only .ko files and userspace binaries—no DKMS, no kernel-devel, no gcc.
Manifest-based userspace installation: Builder installs packages with rpm --nodeps, generates file manifest via rpm -ql, writes to zfs-files.txt. Final image reads manifest and copies only those files via bind mount. Result: zfs, zpool, nvidia-smi without pulling DKMS as a dependency. Most bootc projects either install packages directly (pulling build deps) or copy entire directories blindly.
┌─────────────────────┐
│ ZFS Module Builder │ kernel-devel + zfs-dkms → dkms build
│ (modules/zfs) │ rpm -ql → zfs-files.txt (manifest)
└──────────┬──────────┘
│ COPY .ko + userspace via manifest
↓
┌─────────────────────┐
│ NVIDIA Builder │ kernel-devel + nvidia-dkms → dkms build
│ (modules/nvidia) │ rpm -ql → nvidia-files.txt (manifest)
└──────────┬──────────┘
│ COPY .ko + userspace via manifest
↓
┌─────────────────────┐
│ Final Image │ COPY --from=builders, depmod -a
│ (base/zfs-nvidia) │ No build tools remain
└─────────────────────┘
CentOS Stream 10's bootc image and kernel-devel package are published by independent pipelines and frequently differ in version — either can be ahead. The CI upstream stage detects the available kernel-devel via dnf repoquery and passes the target version (KMOD_KERNEL) to all module builders. Each builder Containerfile compares the image's installed kernel-core against KMOD_KERNEL using rpm.vercmp() and calls dnf downgrade or dnf upgrade as needed to pin the kernel before compiling modules. This ensures kernel-core and kernel-devel always match regardless of which direction the mismatch goes.
| Distro | Mismatch frequency | Notes |
|---|---|---|
| CentOS Stream 10 | High | Decoupled bootc and repo pipelines |
| Fedora 43 | Moderate | Faster churn, shorter lag windows |
| RHEL | Low | EUS pins kernel-devel to kernel-core |
Nightly full rebuilds, no caching. Version detection happens during build (chicken-and-egg for cache keys), fresh signatures on every build simplify provenance. Stages: upstream → modules → base → publish.
┌────────────────────────────────────────────────────────────┐
│ TRIGGER: Schedule (nightly) / Web / Main Push / MR │
└────────────────────────┬───────────────────────────────────┘
┌───────────────┴───────────────┐
▼ ▼
┌──────────────────────┐ ┌──────────────────────┐
│ stream10 (CentOS) │ │ fedora43 │
└──────────┬───────────┘ └──────────┬───────────┘
┌───────┴───────┐ │
▼ ▼ │
┌─────────┐ ┌─────────────┐ │
│kmod-zfs │ │kmod-nvidia │ │
└────┬────┘ └─────┬───────┘ │
└─────┬──────┘ │
┌──────────┼──────────────┐ │
▼ ▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────────┐ ┌────────────┐
│ base │ │base-zfs│ │base-zfs- │ │fedora-base │
│ │ │ │ │nvidia │ │ │
└───┬────┘ └───┬────┘ └─────┬──────┘ └─────┬──────┘
└──────────┴────────────┴──────────────┘
│
▼
┌───────────────┐
│ publish │
│ (badges) │
└───────────────┘
Each base job pushes :latest and signs inline.
Instance repos (Tier 3) include the shared template
and rebuild when base images change.
Validation: Builder stages verify module existence (modinfo), library linking (ldd), manifest completeness. Base stages verify no build deps leaked, single kernel, OCI compliance (bootc container lint). Hardware validation requires physical deployment.
Runner: Rootless podman with overlay storage. Privileged job containers constrained within user namespace—cannot escalate to host root.
All images signed via Sigstore cosign (keyless OIDC through GitLab). CycloneDX SBOMs generated with syft and attested alongside each image. No private keys—signing identity is the CI pipeline itself, logged in Rekor transparency log.
Upstream CentOS/Fedora bootc images are NOT signed by Red Hat/Fedora. This project mirrors and signs them to provide complete supply chain coverage.
Limitations: No Secure Boot support (ZFS CDDL licensing, NVIDIA built from source). Container policy.json verification blocked pending containers/image cosign bundle support—images are signed but deployed hosts can't verify yet.
-
Download and boot Fedora CoreOS in UEFI mode
- Verify UEFI:
ls /sys/firmware/efi
- Verify UEFI:
-
Enable SSH for easier registry authentication:
sudo su rm /etc/ssh/sshd_config.d/40-disable-passwords.conf systemctl restart sshd passwd core
-
Login via SSH (makes copying registry tokens easier):
ssh core@<ip-address> sudo su
-
Authenticate to registry:
podman login registry.gitlab.com
-
Install to disk:
sudo podman run --rm --privileged \ --pid=host \ -v /dev:/dev \ -v /var/lib/containers:/var/lib/containers \ registry.gitlab.com/dunn.dev/immutable-base/base-zfs-nvidia:latest \ bootc install to-disk \ --karg="console=ttyS0,115200n8" \ --karg="nouveau.modeset=0" \ --karg="nvidia-drm.modeset=1" \ --karg="zfs.zfs_arc_max=68719476736" \ /dev/nvme0n1Important: Do NOT use
--generic-imageflag when installing to physical hardware. This flag skips creating EFI boot entries, which will prevent the system from booting. Only use--generic-imagewhen creating portable disk images for distribution.Kernel arguments:
console=ttyS0,115200n8- Serial console for IPMI/BMC accessnouveau.modeset=0- Disable nouveau driver to prevent conflicts with NVIDIAnvidia-drm.modeset=1- Enable NVIDIA DRM kernel mode setting (required for nvidia-smi)zfs.zfs_arc_max=68719476736- Limit ZFS ARC cache to 64GB (adjust based on available RAM)
-
Reboot and remove live media
Instances are separate GitLab projects that consume base images, adding machine-specific configuration (users, SSH keys, service accounts). Each instance has its own CI/CD pipeline, secrets, and update schedule.
All instances share a single CI template located at instance/.gitlab-ci.yml. To adopt it, set your instance repository’s .gitlab-ci.yml to:
include:
- project: "dunn.dev/immutable-base"
ref: v1.0.0
file: "instance/.gitlab-ci.yml"
variables:
BASE_IMAGE: "registry.gitlab.com/dunn.dev/immutable-base/base:latest" # choose variant neededThe template pins cosign v2.6.2 with --new-bundle-format=false, installs syft (and jq), generates CycloneDX SBOMs, uploads them as GitLab cyclonedx artifacts, and attaches signed attestations. Nightly scheduled pipelines only rebuild when the upstream base image config digest changes; merge requests and main-branch commits always rebuild.
Keep BASE_IMAGE aligned with the immutable-base variant your instance requires. Ensure your Containerfile accepts BASE_BOOTC_IMAGE as a build arg. Pin to a release tag (e.g., ref: v1.0.0) for stability; bump the ref when you want to adopt template improvements.
Special thank you to Ben Bread and his blog for demonstrating bootc viability for this use case.
- bootc: https://containers.github.io/bootc/
- OpenZFS: https://openzfs.github.io/openzfs-docs/
- NVIDIA Container Toolkit: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/
- AMD ROCm: https://rocm.docs.amd.com/
- Sigstore cosign: https://docs.sigstore.dev/
- CycloneDX SBOM: https://cyclonedx.org/
- OCI Image Spec: https://github.com/opencontainers/image-spec
- coreos/fedora-bootc-nvidia: https://github.com/coreos/fedora-bootc-nvidia - Original pattern for NVIDIA integration with bootc
- Fedora bootc examples: https://gitlab.com/fedora/bootc/examples - Official examples including kernel module pattern
- mrguitar/fedora-nvidia-bootc: https://github.com/mrguitar/fedora-nvidia-bootc - Modern bootc GPU drivers guide
- mrguitar/bootc-appliance: https://github.com/mrguitar/bootc-appliance - Appliance-style bootc configurations
- mrguitar/rhel9-bootc-demos: https://github.com/mrguitar/rhel9-bootc-demos - RHEL 9 bootc demonstrations
- osbuild/bootc-image-builder: https://github.com/osbuild/bootc-image-builder - Tool for building bootc disk images
This project contains configuration and build scripts for assembling bootc images from various upstream components:
Component Licenses:
- CentOS Stream: GPLv2 and various open source licenses
- OpenZFS: CDDL 1.0 (Common Development and Distribution License)
- NVIDIA Drivers: Proprietary NVIDIA Software License
- bootc: Apache-2.0 OR MIT
- systemd: LGPLv2.1+
This Repository:
- Configuration files, scripts, and documentation: MIT License
- Built images: Governed by component licenses above
License Compatibility: This project uses MIT for its original work (Containerfiles, scripts, CI configuration) which is compatible with all upstream licenses. The MIT license allows redistribution and modification while being compatible with both permissive (Apache, BSD) and copyleft (GPL, LGPL) licenses. Note that CDDL (ZFS) and GPL are generally considered incompatible for static linking, but since ZFS kernel modules are loaded dynamically at runtime, this does not create a licensing conflict.