Smoke tests (first pass, take 2)

[case]

(This PR replaces #316, which was based on my rv repo fork, from before I was granted direct access to the rv repo)

Here's a first stab at an rv clean-install "smoke tests" suite for some popular / reference Ruby projects:

• Diaspora
• Discourse
• Fastlane
• GitLab
• Homebrew
• Huginn
• Lobsters
• Mastodon

If this looks roughly aimed in the right direction, we can iterate and add more projects. (and if not, we can iterate and take a different approach!)

Tl;dr of how it works:

1. Build rv
2. We use a oneoff, temporary Dockerfile for the project
3. Copy our newly built rv binary into the Dockerfile
4. Clone the Ruby project's git repo
5. rv determines the project's Ruby version from Gemfile.lock, courtesy of our work in Use Ruby version from Gemfile.lock as a fallback #325
6. rv installs the project's Ruby version
7. Using the newly installed Ruby, run rv ci to clean-install the project

Here's an overview:

• There's a bin/smoke-tests dir with scripts and Dockerfiles for the various Ruby projects - this is meant to keep the test configs separate from the rest of the codebase
  • Maybe this would be better suited for a top-level /tests dir instead? I am relatively new to Rust, so I'm not sure what the conventions are for stuff like this.
• Dockerfiles for each project - given that we want to run these tests locally and in CI (Linux), I figured a simple Dockerfile approach made sense, to keep things simple and isolated. Happy to revisit if folks have better ideas. We are essentially checking to see, "Can rv install a given project's Gems?" - so we sort of need to hack rv into the projects' individual installation workflows.
  • This required adding a .dockerignore file, in order to keep the Dockerfiles' context and sizes small
• There's a new smoke_tests.yml file, to run these in GH Actions. It will only run via the Actions' web UI's manual trigger button; it isn't needed for most pull requests.

And a few minor, semi-related things that I added along the way:

• Added the Mastodon Gemfile.lock, to improve our tests and parsing coverage for another popular project. This was related to Add Gemfile parsing for git refs and tags #313 which was a side quest from this PR.
• Added local bin/integration-tests/alpine and bin/integration-tests/arch scripts, so that we can run the Linux integration tests locally too

And for fun, I added an idea of a "baseline" test, to do a native-Ruby tooling install as well - this will let us start to see rv's speed benefits.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[deivid-rodriguez]

This is great, just had some minor comments.

I'm afraid all these scripts will tend to get out of date and break if we don't run them often in CI, so it's a pity that discourse is so slow since the rest of the tests seem like they could run on every PR?

Anyways, if we remember to run this on every release, I think that's good enough 👍.

[indirect]

Thanks for getting this set up and working! My main impression is that this is very complicated, but I'm having trouble figuring out why.

• Is Docker helping us with something that we can't do by just running apt-get install in a script?
• GHA runners are already Linux by default and macOS on request, so doesn't that mean we don't need to cross-compile?
• Since we already have build jobs on every commit, is there a reason we can't re-use the existing built rv?
• Why does this add a GHA job to build rv, upload an artifact, and then download that artifact into the smoke test job, when every smoke test script also builds rv directly?
• Cargo already knows how to build and run rv, so why are the smoke-test scripts building rv and then doing architecture conditionals to copy it somewhere rather than just running it directly with our bin/rv script?
• rv already installs ruby as part of rv ci, so why are the Dockerfiles running rv ruby install separately?
• rv already handles the PATH to run its rubies, so why are we manually setting the PATH before calling rv ci?
• if we are running these smoke tests inside ephemeral docker containers or GHA jobs, why are we cleaning up?
• generally speaking, docker build is not the mechanism to run a test, it's the mechanism to create a docker image. I'd prefer we run our tests via docker run if we need to use docker, or just via a regular script if we don't

Zooming way back out, my understanding of the goal here is: 1) collect several example lockfiles, 2) ask rv to install those lockfiles to make sure that they work. Is that right?

If those are the only goals we need to satisfy, I would suggest removing Docker entirely, and just having one script per smoke-test. Then script can be run in a separate GitHub Actions matrix job. I've rebased and added a commit that shows what I mean just for Diaspora. Hopefully that makes sense, but feel free to ask questions!

[case]

Thanks @indirect!

Yeah this was a bit of a meandering journey - I used Docker initially because I knew it would achieve parity between local work (Mac) and CI (Linux), but now that we can see it all working correctly, it makes sense to simplify it.

These changes are pushed now.

[deivid-rodriguez]

To be honest I liked the previous (admittedly more complex) setup, so that I could simulate all these tests locally from macOS.

Why does this add a GHA job to build rv, upload an artifact, and then download that artifact into the smoke test job, when every smoke test script also builds rv directly?

I don't think it built rv directly, but copied the built artifact into the image. I think it was mainly to save cargo install time since it would happen only once for all smoke tests.

rv already installs ruby as part of rv ci, so why are the Dockerfiles running rv ruby install separately?

That's a really good point, I had the same thought when looking into this but finally decided that rv ci needed ruby on PATH for some reason. I don't think that's the case after all, it can shell out to its own managed rubies without them being in PATH, right? I recall some test failures I saw on Linux where some specific ruby processes launched by rv were using system ruby (which I did not have installed), could be related to issues with that, too.

[indirect]

@deivid-rodriguez I've added bin/smoke-test, which we can hopefully use to run the exact same test using Docker on macOS, without needing a separate Dockerfile for each test. Let me know what you think. 😅