WIP: implement a secret vars store in nixpkgs by Lassulus · Pull Request #370444 · NixOS/nixpkgs

Lassulus · 2025-01-03T01:49:04Z

This allows to create secrets (and public files) outside or inside the nix store in a more delclarative way. This is shipped with an example (working) implementation of an on-machine storage. The vars options can easily be used to implement custom backends and extend the behaviour to integrate with already existing solutions, like sops-nix or agenix.

I wanted to upstream this, since we use code similiar to this in https://clan.lol now for a while. Happy to discuss it a bit more. I'm not very fond of the name vars anymore, so maybe a first change could be to come up with a better name :)

I'm not sure if this code is working yet, I will try to develop it a bit more in the next coming days, but I wanted to get this out earlier so other people can throw in some feedback already :)

lucasew · 2025-01-03T23:51:26Z

How this compares to sops-nix?

Lassulus · 2025-01-04T00:06:20Z

How this compares to sops-nix?

This enhances sops-nix, It is more of an upstream evolution of agenix-rekey.

The gist is: This supports declaring how to generate the secrets you store in sops-nix in nixpkgs in the service module.

I will add a declaration to a service soon, so it is easier to understand :)

Lassulus · 2025-01-07T20:39:57Z

alright, removed some features to make the first implementation easier.
Some things are still missing in the on-machine backend like dependencies and prompts. but the rough shape should be more clear now :)

nixos-discourse · 2025-01-08T15:35:18Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2025-01-08-clan-weekly-changelog/58501/1

oddlama · 2025-01-08T22:04:17Z

This looks awesome, I'll have to play around with it a bit more to understand all the details.

Have you already thought about how this would work when someone wants to use vars/secrets in the initrd in stage1? I imagine there could be some kind of hen-egg problem since the initrd will be generated pretty early on when a new generation is activated. The configured vars/secrets may probably not exist at that time, so this might require activating twice (?)

lucasew · 2025-01-08T22:29:23Z

I am curious about how it would work in stage 1 because AFAIK / is not mounted yet in this phase, so how it will read the host key to decrypt the secrets.

Lassulus · 2025-01-09T11:40:25Z

Have you already thought about how this would work when someone wants to use vars/secrets in the initrd in stage1? I imagine there could be some kind of hen-egg problem since the initrd will be generated pretty early on when a new generation is activated.

The current code here doesn't run during activation but creates a script that can be run before activation. in clan.lol we solved that issue differently by having a requiredBy option which can take "activation". We would create the files necessary for activation beforehand in that case, but I think this adds too much complexity to this initial PR.
But this can be something we can add later on top of it.
For now just having all the files created in one phase would be easier and since the creation script can be run at any time (even before installation) they should be available for all phases.

Another problem is also the prompts which some vars can ask for. Having the generation run in the activation would make prompting for input very hard. So I think a better way would be to prompt in nixos-install/nixos-rebuild, so before running activation. But this is also something I will add in a later iteration :)

Lassulus · 2025-01-09T12:55:34Z

I am curious about how it would work in stage 1 because AFAIK / is not mounted yet in this phase, so how it will read the host key to decrypt the secrets.

This is not really in scope for this PR. The current on-machine implementation I added here, doesn't encrypt the secrets (yet?) so there should be no issue with stage-1. secrets can still be added to initrd if they are required to early boot and they should be available at that time.

nixos/modules/system/vars/options.nix

oddlama · 2025-01-12T13:03:18Z

nixos/modules/system/vars/options.nix

Can we add a value option or something similar to each prompt? I would like to be able to pre-define the value of the prompt to ensure that a user can always opt-in to a purely declarative configuration.

Btw, I currently don't understand when an author of a nixos module would decide to utilize a prompt instead of a generated random value. If I am writing a generator for my personal configuration, I can understand that I might want to use prompt for certain things - but when would I want to use a prompt in an upstream module and force interactivity on every user?

I'm just a little worried right now that introducing prompts without a way to declaratively set their value will sooner or later result in some generators being added to nixpkgs which depend on some form of interactivity. Therefore I'd like to have a way to specify the value of a prompt ahead of time, so the generation script can then just copy the specified file to prompts/$name instead of asking the user a question.

we mainly use prompts for API keys, which we usually don't want to leak into the config. I guess we can think about pre declaring things, this would also make the testing code a bit easier which I worked on last time I touched that code. But this increases the surface every backend implementation has to take care of.

oddlama · 2025-01-12T13:51:56Z

nixos/modules/system/vars/options.nix

These are currently not used by the on machine generator. Please ignore if that is still in the works :)

I was thinking of removing them again, since the chmod/etc needs to happen after activation anyway and the user/group has to exist for that to work. But we use those definitions in clan.lol generators every now and then.

If you remove these, everything will be owned by root I presume? That might make it hard to provision secrets for services which don't use LoadCredential

ideally we want to fix that in the preStart or with tmpfiles.d but there could be certain cases where it would be necessary to do it beforehand. But maybe it would be better to have a separate activation-script/service for that, that takes a file somewhere on the filesystem and links/copies it to the target? but tmpfiles also does that already, so not sure

I'm also not sure what would be the best action here, I guess I'm mostly just used to being able to specify it on the secret itself

nixos/modules/system/vars/options.nix

This allows to create secrets (and public files) outside or inside the nix store in a more delclarative way. This is shipped with an example (working) implementation of an on-machine storage. The vars options can easily be used to implement custom backends and extend the behaviour to integrate with already existing solutions, like sops-nix or agenix.

nixos/modules/system/vars/on-machine-backend.nix

jfly

I'm excited about this!

Sorry for the pretty myopic/nitpicky review, hopefully it's more useful than noisy. Here are some more high level thoughts:

The vars options can easily be used to implement custom backends and extend the behaviour to integrate with already existing solutions, like sops-nix or agenix.

I'd be interested to hear more about how this would work. Is the idea that a tool like sops-nix would provide a "backend" (similar to the on-machine backend you've included here?). It seems like the bulk of the logic in this PR is in the generate-vars script in the on-machine backend. Are all backends going to have a similar large chunk of logic (generating inputs, prompting, invoking generators with tempdirs, moving files)?
I think it would be nice to lightly document just what a "backend" is (how to create one). For example, are they all supposed to provide a globally installed generate-vars command?
I personally find the generate-vars shell script to be surprisingly readable, but hard to confirm is robust. I think I saw a few missing shell escapes (I left comments about them). I'd feel more sure about things if we were instead generating a JSON config file for something written in python (or something) to deal with. (Similar to how sops-nix works under the hood if you are familiar with that.)
I'm not very fond of the name vars anymore, so maybe a first change could be to come up with a better name

I personally am not bothered by the word "vars". I assume this isn't called "secrets" because these things sometimes have a secret and a public part (as in your syncthing example)?
I did find the use of the word "var" to be a little confusing while reading the code. Specifically, the relationship between a "var", a "generator" and a "file". IIUC, a "var" has multiple "files" in it, and each file may be secret or public. A "generator" can generate these files. But what's the difference between a var and a generator? They sort of seem to be the same thing. I also saw the word "fact" floating around. I assume that's an old name, or something from clan?

jfly · 2025-01-26T10:03:15Z

nixos/modules/services/networking/syncthing.nix

nit: I'd feel better about this if we somehow asserted that the result of this is exactly 1 ID. I imagine things would get weird if this started returning multiple IDs or no ID at all.

Or, perhaps we could ask upstream syncthing if they'd be open to a change to syncthing generate to persist the id somewhere?

Suggested change

< "$out"/config.xml grep -oP '(?<=<device id=")[^"]+' | uniq > "$out"/syncthing.pub

< "$out"/config.xml grep -m 1 -oP '(?<=<device id=")[^"]+' > "$out"/syncthing.pub

should guarantee that it can yield no more than 1 id (uniq could still yield multiple, different ids). Due to the regex match not applying to empty strings, grep will already return rc 1 if no id is found, so the set -e in the reference backend should fetch it.

Might be a good idea to explicitly define that scripts are supposed to be run with set -e. :)

jfly · 2025-01-26T10:11:36Z

nixos/modules/system/vars/on-machine-backend.nix

nit:

Suggested change

# if not, they are in an incosistent state and we bail out

# if not, they are in an inconsistent state and we bail out

jfly · 2025-01-26T10:20:28Z

nixos/modules/system/vars/options.nix

nit:

Suggested change

Takes the file as an arument and returns maybe an attrset with should at least contain the `path` attribute.

Takes the file as an argument and returns maybe an attrset with should at least contain the `path` attribute.

jfly · 2025-01-26T10:24:29Z

nixos/modules/system/vars/options.nix

"fact" feels like a new piece of vocabulary here. Is this perhaps a term from clan? Should this be "generated file" (or "var")?

Related: what does "public" mean here? Are there also private facts?

jfly · 2025-01-26T10:30:10Z

nixos/modules/system/vars/options.nix

nit: This admonition seems a little bit at odds with the fact that this defaults to true. I personally would expect an option that's documented as "enable this only if ..." to default to false.

idea: Would it be better if we could get rid of this option and instead deduce it based on if the generated file is used as an input to other generators? Right now, I think the API here doesn't allow us to know that (generators depend on other generators). Perhaps it would be better for generators to depend on specific files from other generators? In addition to allowing us to remove this option, it might also move some issues "further left" (such as if generator B depends on generator A and assumes that generator A will create a certain file, but never actually creates the file. We could detect that error when running A, rather than waiting for B to (hopefully) fail.

should be disable. We cannot deduce this automatically since a file can be referenced in another generator and still be needed to be deployed. We also cannot depend on single files by other generators, because files are stored as a unit. For the on-machine backend this option doesn't really make sense though, since we have no other storage location than where the files should be anyway.

We cannot deduce this automatically since a file can be referenced in another generator and still be needed to be deployed.

Sorry, I'm not quite following this sentence: if a file is referenced in another generator, then doesn't that mean that it does need to be deployed?

I think I'm unclear on where the generators are supposed to run. Is it on the target machine, or on some other machine doing the build?

well that depends on the backend that runs the generators. The declarations of the generators inside nixpkgs should be independent of where they run.
What I wanted to say, is that a generator can be a dependency of another generator but the secret file that is depended on can still be required by the system to be present during runtime. for example ssh-ca, where we would generate a certificate and a private key in a generator and later have a ssh key be generated that needs the private key to sign the public key, so it gets trusted automatically. In that case we would need to deploy the certificate from the first generator to the machine and the public key, private key and certificate from the second generator.

OK, I think that makes sense.

I think the part I'm missing is how these generated files actually get "wired up" to running services. Is that present in the syncthing example in this PR? (I don't think it is).

If we have some way of expressing "this file created by this generator is needed by this service" (which feels like something we'd have to do anyway), then wouldn't that let us deduce if the file should be deployed to the machine?

https://github.com/NixOS/nixpkgs/pull/370444/files#diff-fd4c91fd302760b673b3e9b6c8d3474f0ae371005b6be1baff552d0aa7ffe223R639-R640

Ah, that makes sense, thanks! For the record, the reason I was asking this question was because I was trying to understand what mechanism would ensure that services would get restarted when secrets change. In other words, is the vars interface missing something like sops-nix's restartUnits functionality to annotate this relationship, or perhaps some other mechanism to ensure restarts (such as requiring that vars backends always generate "input addressed secret paths").

You, me, @a-kenji, @Kranzes (and others) spent some time discussing this last week. My summary of the conversation:

Generators that store secrets completely "out of band" (such as the on machine backend in this PR) cannot possibly implement something like "input addressed secret paths".

We don't need to expose an additional api similar to sops-nix's restartUnits functionality, because we can already achieve this with systemd functionality.
(set up a unit to watch the file, trigger a restart of our service based on that)

We could propose a patch to systemd to have it automatically watch the files that that LoadCredential points at and restart them.

@Kranzes pointed out that this wouldn't be possible for systemd to do if LoadCredential points at a socket. Still, this does feel like something that should be supported in systemd... somehow.

Generators that store secrets completely "out of band" (such as the on machine backend in this PR) cannot possibly implement something like "input addressed secret paths".

Upon further reflection, I think it's a little more nuanced than this. It's hard to be concrete about this because the precise context in which generators are invoked is not yet specified. But, you could imagine that invocation of generators is done in a way such that we store a "generation id" and are able to inject that id into the string interpolation that produces secret paths. This implies a couple things:

There's some mechanism for storing and fetching the secret "generation id" that is readable at nix evaluation time. Either files in a repo, or module args injected "impurely". This would be a change to the vars api as in this PR.

It would not be possible to rotate secrets completely out of band. You'd have to deploy an updated nixos configuration. Lame, but still better than the "reboot your machine to pull in updated secrets" that we have with the current state of things + this PR.

jfly · 2025-01-28T00:36:46Z

nixos/modules/system/vars/on-machine-backend.nix

Does this need shell escaping? (could file.path have spaces in it?)

jfly · 2025-01-28T00:39:49Z

nixos/modules/system/vars/on-machine-backend.nix

Does this need shell escaping? Could gen.name contain a double quote? I think so.

Not double quotes as currently all names have the type lib.types.strMatching "[a-zA-Z0-9:_\\.-]*", but ending a name with \ would break this line. Quoting using ' would alleviate this, but for robustness sake, escaping most injected values seems preferable to me.

jfly · 2025-01-28T00:46:23Z

nixos/modules/system/vars/on-machine-backend.nix

This '${file.name}' feels like a "poor man's escaping". I'm pretty sure that file.name could itself contain a ', so I don't think this works. Can we use lib.escapeShellArg here instead?

jfly · 2025-01-28T00:48:32Z

nixos/modules/system/vars/on-machine-backend.nix

nit: This is unnecessary because of the trap 'rm -rf $out' EXIT up above, right?

ah, the trap only runs on exit, it's better to clean up the files if they are no longer needed earlier. the trap is just in case the script exists unexpectedly

jfly · 2025-01-28T01:11:35Z

nixos/modules/services/networking/syncthing.nix

I'm not understanding the glue between this generator and the syncthing service. How do the files created by this generator actually get wired up into the running service?

Hope I am not misreading your question, but the pem files are seemingly simply used to set conventional nixos options such as on line 639-641 in this example.

The device id in syncthing.pub does not seem to be used in the example, but only stored. Presumably for outside use just as scripted setups of other devices to sync with.

Lassulus · 2025-02-01T06:55:58Z

1. > The vars options can easily be used to implement custom backends and extend the behaviour to integrate with already existing solutions, like sops-nix or agenix.
   
   
   I'd be interested to hear more about how this would work. Is the idea that a tool like sops-nix would provide a "backend" (similar to the on-machine backend you've included here?). It seems like the bulk of the logic in this PR is in the `generate-vars` script in the on-machine backend. Are all backends going to have a similar large chunk of logic (generating inputs, prompting, invoking generators with tempdirs, moving files)?

yes basically every implementation has to take care of executing the scripts and storing the generated files in the location, also deployment of those files needs to be taken care of by a backend.

2. I think it would be nice to lightly document just what a "backend" is (how to create one). For example, are they all supposed to provide a globally installed `generate-vars` command?

no, the generate-vars is mostly only useful in this usecase, since generating and deploying of files happens at the same time. Otherwise I would do deployment in activation. But we cannot prompt for inputs there. So this was moved into a script that can be manually run at any time.

3. I personally find the `generate-vars` shell script to be surprisingly readable, but hard to confirm is robust. I think I saw a few missing shell escapes (I left comments about them). I'd feel more sure about things if we were instead generating a JSON config file for something written in python (or something) to deal with. (Similar to how sops-nix works under the hood if you are familiar with that.)

sure, you can just dump the vars config from nix with json. But that increases the complexity of the on-machine implementation, I want a simple one that people can read and understand how to adapt. sops-nix probably would interact more directly with the vars instead of shell scripts.

4. > I'm not very fond of the name vars anymore, so maybe a first change could be to come up with a better name
   
   
   I personally am not bothered by the word "vars". I assume this isn't called "secrets" because these things sometimes have a secret and a public part (as in your syncthing example)?
   I did find the use of the word "var" to be a little confusing while reading the code. Specifically, the relationship between a "var", a "generator" and a "file". IIUC, a "var" has multiple "files" in it, and each file may be secret or public. A "generator" can generate these files. But what's the difference between a var and a generator? They sort of seem to be the same thing. I also saw the word "fact" floating around. I assume that's an old name, or something from clan?

yeah, fact is the old name for vars. we can't really call them generators, because that is already used by other parts of nixos. but in the end vars are generators that generate secret or public files :D

Lassulus · 2025-02-01T07:01:02Z

pushed some changes, so some of the nitpicks should be more or less solved now. primarily restricted the allowed characters in certain names, so we can make escaping easier.

a-kenji · 2025-02-01T08:40:28Z

Possible names for vars could possibly be:

vars
keys
facts
more?

Would be happy if someone has more inputs on this.
As already said we already have generators themselves covered as a toplevel concept.

But what this would be is some form of key/var/fact-generator / generating mechanism.

Kranzes · 2025-02-26T11:43:14Z

nixos/modules/system/vars/on-machine-backend.nix

+        elif [ $all_files_missing = true ] ; then
+
+          # prompts
+          prompts=$(mktemp -d)


We should slap umask 077 in places...

4z3 · 2025-03-07T10:30:52Z

nixos/modules/system/vars/on-machine-backend.nix

+
+          # outputs
+          out=$(mktemp -d)
+          trap 'rm -rf $out' EXIT


Traps override each other. Confer (trap 'echo a' EXIT; trap 'echo b' EXIT).

The way it stands, unless the script exits prematurely, only the very final 'rm -rf $out' gets executed, leaving all the other $outs, and all $ins and $promptss unremoved. This issue becomes apparent when declaring variables readonly.

One possible solution would be to spawn a subshell for each generator, e.g.:

... elif [ $all_files_missing = true ] ; then ( prompts=$(mktemp -d) in=$(mktemp -d) out=$(mktemp -d) readonly prompts in out # optional assertion trap 'rm -fR "$prompts" "$in" "$out"' EXIT export prompts in out ... ) fi

It's also possible to have an array of cleanup function: https://github.com/Mic92/dotfiles/blob/4ce52e794985cf159e69e259e8b63d2f7cadfa22/pkgs/merge-when-green/script.sh#L10

4z3 · 2025-03-07T11:17:23Z

nixos/modules/system/vars/on-machine-backend.nix

+          prompts=$(mktemp -d)
+          trap 'rm -rf $prompts' EXIT
+          export prompts
+          mkdir -p "$prompts"


mktemp -d does the mkdiring.

4z3 · 2025-03-07T11:23:20Z

nixos/modules/system/vars/on-machine-backend.nix

Not double quotes as currently all names have the type lib.types.strMatching "[a-zA-Z0-9:_\\.-]*", but ending a name with \ would break this line. Quoting using ' would alleviate this, but for robustness sake, escaping most injected values seems preferable to me.

4z3 · 2025-03-07T11:41:16Z

nixos/modules/system/vars/on-machine-backend.nix

+
+          # move the files to the correct location
+          ${lib.concatMapStringsSep "\n" (file: ''
+            OUT_FILE="$OUT_DIR"/${if file.secret then "secret" else "public"}/${file.generator}/${file.name}


Can we somehow reuse vars.settings.fileModule here? I see how we do want OUT_DIR to be overridable, but reconstructing this path it multiple times feels wrong to me.

4z3 · 2025-03-07T11:47:06Z

nixos/tests/all-tests.nix

  user-home-mode = handleTest ./user-home-mode.nix {};
  ustreamer = handleTest ./ustreamer.nix {};
  uwsgi = handleTest ./uwsgi.nix {};
+  vars = handleTest ./vars.nix {};


This placed seemed pretty ordered until this line got added. Or is all-tests.nix also just kindasorted like module-list.nix :)

4z3 · 2025-03-07T12:06:44Z

Would be happy if someone has more inputs on this. As already said we already have generators themselves covered as a toplevel concept.

Why not? I see that #370444 (comment) says

we can't really call them generators, because that is already used by other parts of nixos

but I fail to see it being used as an option in NixOS by both looking the manual and by trying to define (and use) options.generators. So if there isn't a compelling reason against it, let's snatch that name.. :D

KiaraGrouwstra · 2025-03-17T19:38:14Z

would this method also support templating techniques like https://github.com/Mic92/sops-nix#templates or https://github.com/jhillyerd/agenix-template?

jfly · 2025-03-17T19:53:16Z

would this method also support templating techniques like https://github.com/Mic92/sops-nix#templates or https://github.com/jhillyerd/agenix-template?

@KiaraGrouwstra, I believe the way you'd do this with vars is to define a secret (the "templated secret") that depends on other secret(s). That alone wouldn't give you a templating language like in the examples you've linked to (I think that's out of scope here), but you could then invoke your preferred templating language when generating the "templated secret".

I don't think this PR has support for dependencies between vars. I'm not sure if the plan is to add that to this PR, or do it in a subsequent PR.

phaer

Thank you so much for this, I'd love having such a feature upstream and think this would at the very least be good step forward!

I've left a few comments, but feel free to ignore all of them (@Kranzes comment regarding umasks and @4z3 about traps overriding each other as well as the eval error are the only 3 i noticed, that should be blockers IMO).
I know you've got a lot on your plate but hope you'll continue working on this or let us know if not, so someone could pick this up, @Lassulus

(Personally not too fond of bash as a language for something that entails so much string & path manipulation, but that's the authors choice ;))

phaer · 2025-03-25T11:09:09Z

nixos/modules/services/networking/syncthing.nix

Hope I am not misreading your question, but the pem files are seemingly simply used to set conventional nixos options such as on line 639-641 in this example.

The device id in syncthing.pub does not seem to be used in the example, but only stored. Presumably for outside use just as scripted setups of other devices to sync with.

phaer · 2025-03-25T11:20:15Z

nixos/modules/services/networking/syncthing.nix

Suggested change

< "$out"/config.xml grep -oP '(?<=<device id=")[^"]+' | uniq > "$out"/syncthing.pub

< "$out"/config.xml grep -m 1 -oP '(?<=<device id=")[^"]+' > "$out"/syncthing.pub

should guarantee that it can yield no more than 1 id (uniq could still yield multiple, different ids). Due to the regex match not applying to empty strings, grep will already return rc 1 if no id is found, so the set -e in the reference backend should fetch it.

Might be a good idea to explicitly define that scripts are supposed to be run with set -e. :)

phaer · 2025-03-25T11:26:40Z

nixos/modules/system/vars/on-machine-backend.nix

+        '') (lib.attrValues gen.files)}
+
+        if [ $all_files_missing = false ] && [ $all_files_present = false ] ; then
+          echo "Inconsistent state for generator: {gen.name}"


Suggested change

echo "Inconsistent state for generator: {gen.name}"

echo "Inconsistent state for generator: ${gen.name}"

(unless this some ultra-obscure bash syntax i am not aware of ;)

I sadly think that "Inconsistent state for generator: ${gen.name}" is a pretty bad error message here. I think it should at least try to communicate to the user what exactly is wrong, i.e. which file was found / not found. Ideally it would communicate something actionable as well, i.e:

You can try purging ''${OUT_DIR:-${cfg.fileLocation}}, after making sure it contains nothing you want to keep

I understand that it's annoying to implement in bash, but I am not sure that bash is the right choice of language here in the first place tbh.

roberth

Without an accompanying section in the NixOS manual, you're kind of leaving me guessing here, so I hope my comments make sense.

roberth · 2025-03-31T10:47:34Z

nixos/modules/system/vars/options.nix

+{
+  options.vars = {
+    settings = {
+      fileModule = lib.mkOption {


Maybe

Suggested change

fileModule = lib.mkOption {

allFiles = lib.mkOption {

roberth · 2025-03-31T10:48:10Z

nixos/modules/system/vars/options.nix

+}:
+{
+  options.vars = {
+    settings = {


Not RFC 42. What's the thought behind this name?

I understand it's the provider-specific settings, similar in spirit to how SelfHostBlocks does it, except here it's (IMO correctly) in terms of modules.

roberth · 2025-03-31T10:49:11Z

nixos/modules/system/vars/options.nix

+                A set of files to generate.
+                The generator 'script' is expected to produce exactly these files under $out.
+              '';
+              defaultText = "attrs of files";


Suggested change

defaultText = "attrs of files";

defaultText = literalMD "attrs of files";

Not sure.

roberth · 2025-03-31T10:53:07Z

nixos/modules/system/vars/options.nix

+        default = { };
+      };
+    };
+    generators = lib.mkOption {


This contains both generators (executables), as well as their target files (calls of those executables).
Since those are in a 1:many relationship (potentially), this part of the option tree should be named after the "target files" instead (or whichever term those are described with).

It seems that you have an insofar unnamed domain entity for groups of files that are generated together, and perhaps its name should be put here.

hmm, the idea was, that one generator can create many files, so the files are below the generators in the hierarchy. Files belong to a generator in that they need to be generated by one, not sure where else in the option tree they could live

roberth · 2025-03-31T10:58:49Z

nixos/modules/module-list.nix

Without an accompanying section in the NixOS manual, you're kind of leaving me guessing here, so I hope my comments make sense.

nixos-discourse · 2025-03-31T11:36:07Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/vars-a-framework-for-managing-secrets-and-computed-values/62411/2

nixos-discourse · 2025-04-03T20:25:54Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/vars-a-framework-for-managing-secrets-and-computed-values/62411/6

Lassulus · 2025-04-05T16:55:36Z

there is also a flake now to test this out (or PR against, instead of doing code suggestions here) https://github.com/Lassulus/vars

I will mainly develop this PR there and at a later point bring over the changes made there into this PR again

nixos-discourse · 2025-04-08T16:22:05Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/vars-a-framework-for-managing-secrets-and-computed-values/62411/14

roberth · 2025-04-08T18:30:29Z

nixos/modules/system/vars/on-machine-backend.nix

+      prompt_value=$(cat)
+    '';
+  };
+  generate-vars = pkgs.writeShellApplication {


If I understand correctly, this script may be run on the deployer host, which can be a different platform.

Suggested change

generate-vars = pkgs.writeShellApplication {

generate-vars = deployerPkgs: deployerPkgs.writeShellApplication {

It is then up to the caller to pass a suitable pkgs instance that works for the deployer host.

EDIT: this is of course not the complete solution because the script contents still refer to pkgs. Use a separate file and callPackage to avoid a footgun.

Suggested change

generate-vars = pkgs.writeShellApplication {

generate-vars = { deployerPkgs }: deployerPkgs.callPackage ./generate-vars.nix { inherit cfg; };

As a test case, you could add aarch64-darwin to this and run the checks on darwin.

usually we overwrite pkgs of the system where the vars are defined if the caller architecture is different from the target systems architecture, since this also supports string context references to pkgs and the vars definitons want to live next to the service definitions.

I recommend against that. It blurs the line too much.

You'll be evaluating a NixOS configuration that "runs on" macOS, which is unexpected. It may work out for you anyway, thanks to laziness, but if vars calls someone's code that reads something from the rest of the NixOS config (accidentally or otherwise), they'll get strange errors like "util-linux is not supported on darwin" or worse.

Overriding is a code smell.

Overriding is in general less efficient than evaluating the right thing in one go, especially if it's a NixOS config.

Explicit is good. By making this a proper parameter, readers and module authors have at least an idea of what's going on, and you don't rely on the overriding magic to never hit them in the face.

fricklerhandwerk · 2025-05-06T11:24:38Z

nixos/modules/system/vars/options.nix

+                lib.types.submodule (file: {
+                  imports = [
+                    config.vars.settings.fileModule
+                  ];
+                  options = {
+                    name = lib.mkOption {
+                      type = lib.types.strMatching "[a-zA-Z0-9:_\\.-]*";
+                      description = ''
+                        name of the generated file
+                      '';
+                      readOnly = true;
+                      default = file.config._module.args.name;


Suggested change

lib.types.submodule (file: {

imports = [

config.vars.settings.fileModule

];

options = {

name = lib.mkOption {

type = lib.types.strMatching "[a-zA-Z0-9:_\\.-]*";

description = ''

name of the generated file

'';

readOnly = true;

default = file.config._module.args.name;

lib.types.submodule ({ name, ... }: {

imports = [

config.vars.settings.fileModule

];

options = {

name = lib.mkOption {

type = lib.types.strMatching "[a-zA-Z0-9:_\\.-]*";

description = ''

name of the generated file

'';

readOnly = true;

default = name;

This seems to be a much more idiomatic way of writing it. That piece of code had confused the hell out of me, but maybe I'm just used to different things.

Only changing it to default = file.name would work to the same effect, but I still find it unusual.

fricklerhandwerk · 2025-05-06T11:43:48Z

nixos/modules/system/vars/options.nix

+                This name will be used to refer to the generator in other generators.
+              '';
+              readOnly = true;
+              default = generator.config._module.args.name;


Suggested change

default = generator.config._module.args.name;

default = generator.name;

nixos-discourse · 2025-05-28T02:21:37Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/zurich-25-05-zhf-hackathon-report/64882/1

nixos-discourse · 2025-09-04T01:53:57Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/is-it-possible-to-use-nix-in-a-project-but-without-putting-all-the-code-in-nix-store/68873/8

KiaraGrouwstra · 2026-01-04T19:18:22Z

if i understood correctly (?), clan's motivation for vars involved cross-node communication as facilitated by e.g. exports, which wasn't viable for implementations within the scope of nixpkgs. as such, this PR may not see updates any time soon.

at the same time, the contracts PR (#432529) offers tooling around handling of secrets agnostic to storage back-ends as well, raising questions on their potential integration.

Lassulus requested a review from oddlama January 3, 2025 01:49

github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Jan 3, 2025

Lassulus force-pushed the nixos-vars branch from 4a478cc to e027034 Compare January 7, 2025 20:38

github-actions bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Jan 7, 2025

Lassulus force-pushed the nixos-vars branch 3 times, most recently from 3f0acbe to 339d2bc Compare January 7, 2025 23:43

Lassulus force-pushed the nixos-vars branch 3 times, most recently from 46b881c to b6b0bed Compare January 12, 2025 00:50

oddlama reviewed Jan 12, 2025

View reviewed changes

Lassulus force-pushed the nixos-vars branch from b6b0bed to fc64ee4 Compare January 13, 2025 10:49

Mic92 reviewed Jan 17, 2025

View reviewed changes

nixos/modules/system/vars/on-machine-backend.nix Outdated Show resolved Hide resolved

h7x4 added 8.has: module (new) This PR adds a module in `nixos/` 8.has: tests This PR has tests labels Jan 17, 2025

oddlama mentioned this pull request Jan 27, 2025

generator.dependencies as a list feels awkward in some cases oddlama/agenix-rekey#69

Closed

jfly reviewed Jan 28, 2025

View reviewed changes

WIP: nixos/syncthing: manage secrets with vars

e8a6fef

Lassulus force-pushed the nixos-vars branch from fc64ee4 to e8a6fef Compare February 1, 2025 07:00

Kranzes reviewed Feb 26, 2025

View reviewed changes

4z3 reviewed Mar 7, 2025

View reviewed changes

phaer reviewed Mar 25, 2025

View reviewed changes

roberth reviewed Mar 31, 2025

View reviewed changes

wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 2, 2025

roberth reviewed Apr 8, 2025

View reviewed changes

KiaraGrouwstra mentioned this pull request May 3, 2025

templated vars Lassulus/vars#1

Closed

fricklerhandwerk reviewed May 6, 2025

View reviewed changes

KiaraGrouwstra mentioned this pull request May 18, 2025

password-store backend Lassulus/vars#3

Open

aciceri mentioned this pull request Jul 11, 2025

Find out how agenix-rekey works and if can work together with agenix-shell aciceri/agenix-shell#50

Open

PatrickDaG mentioned this pull request Aug 6, 2025

fix(flake-module): handle missing homeConfigurations oddlama/agenix-rekey#112

Merged

PatrickDaG mentioned this pull request Sep 5, 2025

How to use agenix-rekey to set env var for nushell through home-manager? oddlama/agenix-rekey#106

Open

jfly mentioned this pull request Sep 12, 2025

nixos/miniflux: add options for all secret files #429983

Closed

13 tasks

	< "$out"/config.xml grep -oP '(?<=<device id=")[^"]+' \| uniq > "$out"/syncthing.pub
	< "$out"/config.xml grep -m 1 -oP '(?<=<device id=")[^"]+' > "$out"/syncthing.pub

	# if not, they are in an incosistent state and we bail out
	# if not, they are in an inconsistent state and we bail out

	Takes the file as an arument and returns maybe an attrset with should at least contain the `path` attribute.
	Takes the file as an argument and returns maybe an attrset with should at least contain the `path` attribute.

	echo "Inconsistent state for generator: {gen.name}"
	echo "Inconsistent state for generator: ${gen.name}"

	defaultText = "attrs of files";
	defaultText = literalMD "attrs of files";

	generate-vars = pkgs.writeShellApplication {
	generate-vars = deployerPkgs: deployerPkgs.writeShellApplication {

	generate-vars = pkgs.writeShellApplication {
	generate-vars = { deployerPkgs }: deployerPkgs.callPackage ./generate-vars.nix { inherit cfg; };

	default = generator.config._module.args.name;
	default = generator.name;

Uh oh!

Conversation

Lassulus commented Jan 3, 2025

Uh oh!

lucasew commented Jan 3, 2025

Uh oh!

Lassulus commented Jan 4, 2025

Uh oh!

Lassulus commented Jan 7, 2025

Uh oh!

nixos-discourse commented Jan 8, 2025

Uh oh!

oddlama commented Jan 8, 2025

Uh oh!

lucasew commented Jan 8, 2025

Uh oh!

Lassulus commented Jan 9, 2025

Uh oh!

Lassulus commented Jan 9, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lassulus Jan 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jfly left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lassulus Jan 12, 2025 •

edited

Loading

4z3 commented Mar 7, 2025 •

edited

Loading