Stuck on "sending inputs" with Nix 2.32

[mweinelt]

I think this isn't doing good to the queue runner. Builds tend to get stuck on "sending inputs" forever. So we're filling slots by these over time, with the throughput getting lower and lower.

I see hydra upgraded and nix with it (2.29 -> 2.32), so these would be the primary suspects.

Originally posted by @vcunat in NixOS/infra#858 (comment)

[mweinelt]

We went from here to there and the two top-most commits touch the queue-runner.

❯ git log 274027eb504c7fe090e00c16fd94f4b832981095..0414ae64ebeea4ca0121515bdff42d7a3869862b --oneline --no-merges --no-decorate 
449791b1 Upgrade Nix to 2.32
58846b0a Deduplicate protocol code more with `ServeProto::BasicClientConnection`
a4990638 bump to nix/nix-eval-jobs 2.31
d36b943e Skip content-addressing test for now
4b2d60e1 bump to nix/nix-eval-jobs 2.30
06be6034 flake.lock: Update
7fa3da75 hydra-plugins: replace jq with perl's own canonical json output

449791b 58846b0

[mweinelt]

We did roll back for now.

[mweinelt]

@Ericson2314 suggested to "try 94eaad2 to master tip, following first parent":

• 94eaad2 (Nix 2.31 bump)
• 58846b0 (Deduplicate protocol code more with ServeProto::BasicClientConnection)

[Ericson2314]

@Mic92 just deployed 94eaad2; we are watching https://hydra.nixos.org/machines

[Mic92]

NixOS/infra#880

[vcunat]

Looks fixed then.

[Mic92]

Now yet. Now we need to try upgrading to newer commits and check where it falls over.

[Ericson2314]

OK Next there are two options to try:

• f1463d4 is the next commit which is almost surely fine

• Revert "Deduplicate protocol code more with ServeProto::BasicClientConnection #1536 is Nix 2.32, but with the commit I think might have caused the problem reverted. But the problem could be something else, like Nix 2.32 itself.