-
-
Notifications
You must be signed in to change notification settings - Fork 159
[RFC 0197] package sets by-name #197
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
4801546
9573e0f
bb11e6a
625449c
1d7fb7f
2c7fc9f
b0c8c39
a2853b8
84b5686
c592d41
ee5f96d
d47b2ad
882c21a
8f16dc0
b26025e
5c95506
2775b53
53f107c
3c4cbdd
2cf527e
fa81fb3
d0bb18b
329891e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,254 @@ | ||
| --- | ||
| feature: nixpkgs package sets by-name | ||
| start-date: 2026-01-24 | ||
| author: quantenzitrone | ||
| co-authors: (find a buddy later to help out with the RFC) | ||
| shepherd-team: (names, to be nominated and accepted by RFC steering committee) | ||
| shepherd-leader: (name to be appointed by RFC steering committee) | ||
| related-issues: https://github.com/NixOS/nixpkgs/pull/482538 | ||
| --- | ||
|
|
||
| # Summary | ||
| [summary]: #summary | ||
|
|
||
| Package sets like `fishPlugins` or `python3Packages` move to a `pkgs/by-name` like structure in | ||
| `pkgs/sets-by-name/<setname>`. | ||
|
|
||
| This doesn't apply to package sets that are auto-generated like `haskellPackages`. | ||
|
|
||
| # Motivation | ||
| [motivation]: #motivation | ||
|
|
||
| - get rid of the package categories as directories (decided in RFC 140 and 146) | ||
| - bring the benefits of by-name to package sets | ||
| - merge bot maintainer merging | ||
| - scaleability | ||
| - isolation | ||
| - vetting | ||
| - *your goal here* | ||
|
|
||
| # Detailed design | ||
| [design]: #detailed-design | ||
|
|
||
| - Create a new directory under `pkgs/`, e.g. `pkgs/sets-by-name` that contains package sets. | ||
| - For example `pkgs/sets-by-name/fishPlugins`, `pkgs/sets-by-name/python3Packages`. | ||
| - Each package set is sharded like `pkgs/by-name`. | ||
| - The following additional have to exist for each package set. | ||
| - `functions.nix`: definitons of functions, like `buildFishPlugin`, `buildPythonPackage`. | ||
| - `overrides.nix`: overrides of packages, like `top-level/all-packages.nix` currently functions as | ||
| an overlay for `by-name` packages. | ||
| - This is something we try to keep empty. Most, maybe all, overrides can be inlined in the | ||
| package. | ||
| - `aliases.nix`: aliases for aliases in package sets behind `optionalAttrs config.allowAliases` | ||
| (like `top-level/aliases.nix`). | ||
| - All package sets with their sharded packages, overlayed with their functions, overrides and | ||
| aliases are automatically called by `top-level/package-sets-by-name.nix`. | ||
| - Versioned package sets like `python316Packages` are done in `all-packages.nix` by overriding the | ||
| default version package set. | ||
| - Versioned package sets without a default version will have to override the default version with | ||
| an error. | ||
| - e.g. `nextcloud*Packages` are in `sets-by-name/nextcloudPackages` and thus autocalled by | ||
| `top-level/package-sets-by-name.nix`, however we will have an alias in `top-level/aliases.nix` | ||
| that says | ||
| ```nix | ||
| { | ||
| nextcloudPackages = throw "Please use nextcloudPackages for a specific nextcloud ersion e.g. nextcloud32Packages."; | ||
| } | ||
| ``` | ||
| ``` | ||
| pkgs | ||
| ├── by-name | ||
| │ └── ... | ||
| ├── sets-by-name | ||
| │ ├── fishPlugins | ||
| │ │ ├── as | ||
| │ │ │ ├── async-prompt | ||
| │ │ │ ... └── package.nix | ||
| │ │ ├── au | ||
| │ │ ... | ||
| │ │ ├── z_ | ||
| │ │ │ └── z | ||
| │ │ │ └── package.nix | ||
| │ │ ├── aliases.nix | ||
| │ │ ├── functions.nix | ||
| │ │ └── overrides.nix | ||
| │ │ | ||
| │ ├── python3Packages | ||
| │ │ ├── a2 | ||
| │ │ │ └── a2wsgi | ||
| │ │ │ └── package.nix | ||
| │ │ ├── aa | ||
| │ │ ... | ||
| │ │ ├── zx | ||
| │ │ │ ├── zxcvbn | ||
| │ │ │ │ └── package.nix | ||
| │ │ │ ├── zxcvbn-rs-py | ||
| │ │ │ │ └── package.nix | ||
| │ │ │ └── zxing-cpp | ||
| │ │ │ └── package.nix | ||
| │ │ ├── aliases.nix | ||
|
Comment on lines
+83
to
+90
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So, a few files mixed within the shard list? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The alternative would be putting them in
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Another alternative is to have one more level inside the package set before the shards ( Which would also make more manageable packagesets that benefit from having a few more support files inside the package set but not as a packageset package. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That might require us to filter out that directory from the autocaller function, but it's a good idea as well.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i think they meant
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Well, it is a matter of taste if
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
So I was thinking like this could happen: By name is used in the case so that way you don't get confused easily with two by-names.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you look at the POC implementation, you can also see that buildFishPlugin is in
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't understand that. This mixes shards and non-shards under
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Well the ideally in the packages it's another set then there would be default.nix which would hook it up.
Probably 👀 |
||
| │ │ ├── functions.nix | ||
| │ │ └── overrides.nix | ||
| │ ... | ||
| └── top-level | ||
| ├── all-packages.nix <- calls all versioned package sets | ||
| ├── by-name-overlay.nix <- used to autocall sharded packages (no change required) | ||
| ... | ||
| └── package-sets-by-name.nix <- autocalls all sets by name | ||
| ``` | ||
| Proof-Of-Concept implementation in https://github.com/NixOS/nixpkgs/pull/482538 | ||
| # Examples and Interactions | ||
| [examples-and-interactions]: #examples-and-interactions | ||
| TODO | ||
quantenzitrone marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| # Drawbacks | ||
| [drawbacks]: #drawbacks | ||
| - only allows top level package sets | ||
| - *your drawback here* | ||
| # Alternatives | ||
| [alternatives]: #alternatives | ||
| <details> | ||
| <summary> | ||
| ## Idea 2: nested by-name structure | ||
| </summary> | ||
| outdated Proof-Of-Concept implementation in https://github.com/NixOS/nixpkgs/pull/483432 | ||
| ### Detailed design | ||
| - Idea 1, but sets are in the existing `pkgs/by-name` structure instead of `pkgs/sets-by-name`, e.g. | ||
| `fishPlugins.puffer` would be in `by-name/fi/fishPlugins/pu/puffer`. | ||
| - Additionally a marker is required in order to distinguish package sets from simple packages, | ||
| such as using a `.packageset` file (example: `by-name/fi/fishPlugins/.packageset`). | ||
| If not the `package.nix` must exist and is called as a package. | ||
| ``` | ||
| pkgs | ||
| └── by-name | ||
| ├── 0_ | ||
| ... | ||
| ├── fi | ||
| │ ├── fiano | ||
| │ ├── fiche | ||
| │ ├── ... | ||
| │ ├── fishnet | ||
| │ ├── fishPlugins | ||
quantenzitrone marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| │ │ ├── .packageset | ||
| │ │ ├── as | ||
| │ │ │ └── async-prompt | ||
| │ │ ├── au | ||
| │ │ ... | ||
| │ │ └── z_ | ||
| │ │ └── z | ||
| │ ├── fishy | ||
| │ ... | ||
| ... | ||
| ``` | ||
| ### Advantages | ||
| - no new directory, just extend `pkgs/by-name` | ||
| - allows nested package sets | ||
| ### Drawbacks | ||
| - `lixPackages` (behind all `lib*` packages) will not be accessible through GitHubs UI | ||
| - having package sets in `pkgs/by-name` may not fit the spirit of RFC 140 | ||
| - *your drawback here* | ||
| </details> | ||
| <details> | ||
| <summary> | ||
| ## Idea 3: package sets in `pkgs/by-name` | ||
| </summary> | ||
| Proof-Of-Concept implementation in https://github.com/NixOS/nixpkgs/pull/483128 | ||
| ### Detailed design | ||
| - Instead of `by-name/<shard>/<pname>` we have `by-name/<shard>/<attrpath>`, so e.g. | ||
| `fishPlugins.puffer` would go in `by-name/fi/fishPlugins.puffer`. | ||
| - The `top-level/by-name-overlay.nix` will call all folders in a `<shard>` that contain a dot as a | ||
| package set. | ||
| ``` | ||
| pkgs | ||
| └── by-name | ||
| ├── 0_ | ||
| ... | ||
| ├── fi | ||
| │ ├── fiano | ||
| │ ├── fiche | ||
| │ ├── ... | ||
| │ ├── fishnet | ||
| │ ├── fishPlugins.async-prompt | ||
| │ ├── fishPlugins.autopair | ||
| │ ├── fishPlugins.z | ||
| │ ├── fishy | ||
| │ ... | ||
| ... | ||
| ``` | ||
| ### Advantages | ||
| - no new directory, just extend `pkgs/by-name` | ||
| - allows nested package sets | ||
| ### Drawbacks | ||
| - huge shards due to package sets | ||
| - currently only few shards like `li` are too large for GitHubs UI, but with this idea more shards | ||
| will be huge as well | ||
| - specifically 12 more shards `em`, `gn`, `ha`, `ho`, `oc`, `pe`, `py`, `rp`, `sb`, | ||
| `te`, `ty`, `vi` and `vi` (for `emacsPackages`, `gnomeExtensions`, `haskellPackages`, | ||
| `home-assistant-component-tests`, `ocamlPackages`, `pearlPackages`, `python3Packages`, | ||
| `rPackages`, `sbclPackages`, `texlivePackages`, `typstPackages` and `vimPlugins`) could become | ||
| inaccessible. | ||
| - some package sets like `lixPackages` (behind all `lib*` packages) will not be accessible through | ||
| GitHub UI | ||
| - having package sets in `pkgs/by-name` may not fit the spirit of RFC 140 | ||
| - it's called pkgs/by-**name** and not pkgs/by-**attrpath** | ||
| - directory names as attrpaths is sketchy | ||
| - unresolved questions | ||
| ### Unresolved Questions | ||
| - How do we handle functions like `fishPlugins.buildFishPlugin`? | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (Answering because this applies to idea 1) I never thought it made sense for functions and derivation builders to be inside package sets. I think our Go infrastructure has the right idea: move them to the top-level, and version them in the attribute name.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This proposal is awkward with Python where you have CPython and PyPy and then language version-dialects within them, and even worse for Common Lisp where you have same base language but multiple compatible implementations with their completely own versioning.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i also think keeping functions in package sets is fine |
||
| - How do we handle aliases? | ||
quantenzitrone marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - How do we handle versioned package sets? | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (Answering because this applies to idea 1) I generally think it's a good idea to think of Nixpkgs as having three types of package infrastructure systems:
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. i want handle both 2 and 3 with this RFC |
||
| - How do we move large package sets? | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (Answering because this applies to idea 1) Some of them are easy enough. If we resolve the aforementioned issues with the python edge cases, it should be a simple matter of moving and renaming files. OCaml should also be pretty simple. The generated sets, like Haskell, Node, Lisp and R, will be impossible, though. Those packages don't actually exist as separate files in Nixpkgs, and are instead imported from some other upstream. This is something that again, we'll have to get the input of package set maintainers on. Moving everything over will end up being a case-by-case basis. |
||
| </details> | ||
| # Prior art | ||
| [prior-art]: #prior-art | ||
| - `by-name` stucture for `python3Packages` https://github.com/NixOS/nixpkgs/pull/449896 https://github.com/NixOS/nixpkgs-vet/pull/180 | ||
| - https://github.com/NixOS/nixpkgs/issues/482537 | ||
| - https://github.com/NixOS/nixpkgs/issues/432625 | ||
| - `tclPackages` has their own `by-name` structure https://github.com/NixOS/nixpkgs/pull/344716 | ||
| - attempt to move `nushellPlugins` to `by-name` https://github.com/NixOS/nixpkgs/pull/482961 | ||
| # Unresolved questions | ||
| [unresolved]: #unresolved-questions | ||
| - Does the handling of versioned package sets work like this? | ||
| - How do we move large package sets? | ||
| - *your question here* | ||
| # Future work | ||
| [future]: #future-work | ||
| What future work, if any, would be implied or impacted by this feature without being directly part of the work? | ||
This comment was marked as off-topic.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updates across package sets are very different. I don't think this should be a goal of this RFC, but instead something that can be developed separately as an
updateScriptthatnixpkgs-updatecan pick up.