Support content caching

[cgwalters]

In ostreedev/ostree#1462

+  # TODO: enhance papr to have caching, a bit like https://docs.travis-ci.com/user/caching/
+  - curl -Lo fedora-atomic-host.qcow2 https://getfedora.org/atomic_qcow2_latest

Something like:

- inputs_url_immutable:
   - https://getfedora.org/atomic_qcow2_latest

Where PAPR would do the fetch, and here we want to follow the Location tag; what the "immutable" above hints at is that we cache based on the final filename.

This would end up in the container as:

/srv/inputs/atomic_qcow2_latest -> Fedora-Atomic-27-20180226.0.x86_64.qcow2

or so?

Obviously the other big caching case is rpm-md repos...maybe in some case we may want to pull container images too. I feel like those need their own separate design, but since we have a major need for the qcow2s I'd like to do that first.

[jlebon]

I think we need to talk in terms of pre and post-OCP timelines. I don't know when the PACI work that's been going on will be ready, but features that could be very differently implemented in OCP vs the current infra I think is worth evaluating whether it can wait.

E.g. for this, once we move to OCP and make VM-in-containers feel more native, I think we can abstract this away completely, right?

[jlebon]

Another way to see this: I think the goal is to make host and cluster provision VMs in containers natively, right? So e.g. YAML files that today have:

host:
  distro: fedora/27/atomic

will just keep working.

We can look at how we want to support BYOVMI (bring-your-own-VM-image :)), or maybe it's just a matter of adding a Dockerfile to the PACI repo to "add your own distro"? Then, the platform can take care of keeping those up-to-date as necessary.

[jlebon]

or maybe it's just a matter of adding a Dockerfile to the PACI repo to "add your own distro"?

Now that we have container: pretty much down in the new infra, I've been looking to implement host:. The obvious first item to tackle is images. I think we want something like what cockpit does, see e.g. cockpit-project/cockpit#9002 and https://209.132.184.69:8493/. We've been burned quite a few times with image updates blocking our PRs in various projects.

Strawman:

• service/Jenkins jobs regularly build container images with baked qcow2 for each distro we monitor
• keep a manifest in this repo (or paci? or maybe we should merge them...) of distro --> container image sha256 that PAPR uses to get the image
• on new container builds, bot pushes to repo to update sha256 of distro
• if any project encounters a severe regression, human reverts bot commit while we fix issue

[jlebon]

The assumption I'm making in that strawman up there is that it's better to shuffle the images around inside containers instead of keeping a store elsewhere like Cockpit does. Mostly because it's easy to do and aligns better with OCP operations. One downside though is that if we kept them separate, we could optimize cluster cases like:

cluster:
  hosts:
    - name: myhost
      distro: fedora/27/atomic
  container:
    image: registry.fedoraproject.org/fedora:27

so that it runs in a single container (not just pod) to run both the VM and the container workload. Though I think we can keep that as an optimization in the case where the base of the container image containing the qcow2 matches the container we want. If we shove CentOS-based qcow2s in CentOS container images and similarly for Fedora, I think we'd probably hit that optimization often.