add bats based tests

[adrian-gierakowski]

to make sure sandboxing works as expected

these run as part of top level om ci, or can be ran individually with:

nix develop ./tests -c ./tests/test.bats

the reason I've added this is that:

1. I want to make sure it works as expected before I start using it
2. I would like to add support for macos with sandbox-exec

[srid]

Can we make this a nix flake check to obviate omnix's steps.custom?

This may be useful

https://github.com/juspay/vira/blob/7142be73fd97586ee485e5e2fec20c06de1af951/nix/modules/flake-parts/tests.nix#L6

[adrian-gierakowski]

Can we make this a nix flake check to obviate omnix's steps.custom?

This may be useful

https://github.com/juspay/vira/blob/7142be73fd97586ee485e5e2fec20c06de1af951/nix/modules/flake-parts/tests.nix#L6

I'll try. I assumed this sandboxing functionality would not work inside nix sandbox :)

[srid]

You may need sandbox = relaxed in nix.conf (which is what my CI uses)

[adrian-gierakowski]

You may need sandbox = relaxed in nix.conf (which is what my CI uses)

yeah, although I think it's better to use impure-derivations

[adrian-gierakowski]

btw. what do you use to format nix code?

[srid]

yeah, although I think it's better to use impure-derivations

Sure

btw. what do you use to format nix code?

nixpkgs-fmt

[adrian-gierakowski]

yeah, although I think it's better to use impure-derivations

Sure

got it working with nix build and nix flake check directly in the subflake, but running via omnix I get:

error: pure derivation '/nix/store/dpyp820z4ly5mcwz641qnjfm9if3j56j-devour-output.json.drv' depends on impure derivation '/nix/store/a1iqfbk7f0h0khaj94lbp751lv021c7s-tests.drv'
Error: `nix build` failed; exit code: Some(1)

We'd need to set __impure = true on devour-output as well. Any good way to do this?

btw. what do you use to format nix code?

nixpkgs-fmt

ok, I'm getting diff on file I didn't change, should I format then in a separate commit?

[adrian-gierakowski]

yeah, although I think it's better to use impure-derivations

Sure

got it working with nix build and nix flake check directly in the subflake, but running via omnix I get:

error: pure derivation '/nix/store/dpyp820z4ly5mcwz641qnjfm9if3j56j-devour-output.json.drv' depends on impure derivation '/nix/store/a1iqfbk7f0h0khaj94lbp751lv021c7s-tests.drv'
Error: `nix build` failed; exit code: Some(1)

We'd need to set __impure = true on devour-output as well. Any good way to do this?

ok, so we could check if any of the checks or packages injested by devour-flake has __impure == true and if so, make devour-output.json impure as well. I can open a PR

[srid]

cool

[adrian-gierakowski]

cool

https://github.com/srid/devour-flake/pull/41/files

[srid]

Could you merge from master? That should use the new devour-flake in CI.

Note: no need for rebase in any PRs on my repos; I generally do a squash merge anyway.

[adrian-gierakowski]

@srid all done, vira ci -b passes locally

[adrian-gierakowski]

Issues with CI same as on master. Tests did succeed locally

[adrian-gierakowski]

something is broken with CI on your repo, but I managed to run on my fork. Unfortunately there are multiple layers of failures:

1. the new tests fail in the sandbox but there are no useful logs

2. ever when running the old tests only, it fails on gh-signoff. I guess this could be cause it's running on a fork?

Running outside the sandbox (with nix develop ./tests -c ./tests/test.bats) works, so I think I'll do that for now. Particularly that I don't know how to conditionally build aarch64-darwin depending on the runner when using vira

[adrian-gierakowski]

signoff failed in this repo as well. Disabled it for now.

[adrian-gierakowski]

@srid CI is finally green.

[srid]

Ah, right. I've removed it 6c97965

[adrian-gierakowski]

Ah, right. I've removed it 6c97965

ok, so I will try to add it to vira.hs and see if your runner will manage to build it. You'll need to add:

extra-experimental-features = impure-derivations ca-derivations

[adrian-gierakowski]

Ah, right. I've removed it 6c97965

ok, so I will try to add it to vira.hs and see if your runner will manage to build it. You'll need to add:

extra-experimental-features = impure-derivations ca-derivations

done

[adrian-gierakowski]

not seeing any CI checks, but maybe you can see then on yours side?

[srid]

Does vira ci -b run successfully on your local Linux box? Because ths is what I see:

[..]
copying path '/nix/store/122dqbvxbvj44gqp7g5w09fk56f43njq-bats-1.12.0' from 'https://cache.nixos.org'...
building '/nix/store/awvggiqf2a91mrzhbx19pnxmkm01qhid-nix-shell.drv'...
building '/nix/store/2s0pjl7fbdygkwj6qkcslnsxjs7jgil6-tests.drv'...
nix-shell> Running phase: buildPhase
error: Cannot build '/nix/store/2s0pjl7fbdygkwj6qkcslnsxjs7jgil6-tests.drv'.
       Reason: builder failed with exit code 1.
error: Cannot build '/nix/store/w6dvakaf050jfl4bwgw8qrngqkwbrp77-devour-output.json.drv'.
       Reason: 1 dependency failed.
❌  Task failed with exit code 1 {cmd=/nix/store/x11zi16bhr30l7sb89jbjr0v1b5hyv87-nix-2.32.4/bin/nix build /nix/store/0ihykf5ilwihmcgbilaxgiic6fph2hli-devour-flake#default -L --print-out-paths --no-write-lock-file --override-input flake ./tests --override-input flake/landrun-nix . --out-link ./tests/result}
10:25 ❌ ERROR [🛞 ;19] Process failed: ExitFailure 1
10:25 ❌ ERROR [🛞 ;19] CI pipeline failed

[srid]

not seeing any CI checks, but maybe you can see then on yours side?

For PRs, I need to run manually until we have juspay/vira#292 in place

[adrian-gierakowski]

Does vira ci -b run successfully on your local Linux box?
yes

[adrian-gierakowski]

not seeing any CI checks, but maybe you can see then on yours side?

For PRs, I need to run manually until we have juspay/vira#292 in place

Cool, btw. would love to learn more about vira. Could it be used instead of Hercules CI? Does it support "effects", or only does nix build?