Distributed builds / multi-node

[srid]

Allow building on more than 1 machine.

We could use

1. Nix store distributed builds, or
2. om ci run --on, or
3. Plain ssh ...

for this.

Braindump

(3) is probably more suitable, because we can run all steps including attic push (#29) when the build happens on remote machine, but then that would require those machines to have deps (om, attic, cachix, git, etc.) installed, ideally at pinned versions. So, (2) is probably the way to go -- it can also happen a step granular level (only om stage is done remotely), but then that would unnecessarily download files locally. So perhaps we should re-evaluate (3) while tackling the dependency issue?

(3) Remote deps

om, attic, git, etc. could be setup by copying their derivations (including pinned nixpkgs). Could just copy the vira flake, and then use a devShell from it containing these tools in PATH.

Then we need a concept of workspace dir for remote nodes too. Umm.

Tasks

• Add 'remote builder' information to UI #218
• Allow building for multiple systems #228
• Implement proper log streaming #285
• Replace nix-native remote builder with explicit remote building (over ssh; à la om ci run --on)
• ...

[shivaraj-bh]

In a Juspay internal team that doesn’t use NixOS, followed https://nix.dev/manual/nix/2.28/advanced-topics/distributed-builds.html#requirements to add remote builders, enabling Multi-node support.

[shivaraj-bh]

In a Juspay internal team that doesn’t use NixOS, followed https://nix.dev/manual/nix/2.28/advanced-topics/distributed-builds.html#requirements to add remote builders, enabling Multi-node support.

This is not reliable. Recently observed:

error: cannot add path '/nix/store/1zmsaibmvpk8rvk66afqk3szz0jlsa8b-common-source-0.0.1.0' because it lacks a signature by a trusted key

Edit: This was fixed by adding the remote-builder user to the trusted-users

[srid]

In a Juspay internal team that doesn’t use NixOS, followed https://nix.dev/manual/nix/2.28/advanced-topics/distributed-builds.html#requirements to add remote builders, enabling Multi-node support.

How is that working out so far?

One downside I can imagine is that it would copy derivations to local store needlessly.

[shivaraj-bh]

In a Juspay internal team that doesn’t use NixOS, followed https://nix.dev/manual/nix/2.28/advanced-topics/distributed-builds.html#requirements to add remote builders, enabling Multi-node support.

How is that working out so far?

One downside I can imagine is that it would copy derivations to local store needlessly.

Yes, that is indeed a problem. A lot of back and forth copying increases the time taken to build.

(3) from the description is what the team asked for as well. But [IIUC] it doesn’t look simple to implement.

Just a thought for (3): Vira could support --on? Instead of copying the results back (time-consuming and often unnecessary given the cache), it could just sync the acid-state of master with the builder's. However, I think it might complect Vira.

[srid]

Option (4)

Simply run separate Vira instances on those platforms. They operate independently, and update CI status on GitHub or Bitbucket. Concretely, #110 + #111 + gh-status of #96 (as a final stage/step) should automatically update status on GitHub PRs for both platforms, and we configure merges to be blocked until both succeed.

[srid]

For a SSH-based approach, this may be relevant:

[srid]

We certainly should monitor the builders too; like 'Tools' page, but in its own dedicated page (since build machines != tools).

From #207

@shivaraj-bh - For monitoring health of builder nodes - Proposed idea: Scan home-manager configured builders in ~/.config/nix/machines and list them (with health) in Tools page.

If we do this, however, it would force users to use home-manager, and that Vira will not be supported through a simple nix run, would it not?

[srid]

OTOH, it may not be an unreasonable compromise to support multi-nodes through home-manager configuration.