Skip to content

lib: Create a generic property testing framework#212858

Closed
infinisil wants to merge 1 commit intoNixOS:masterfrom
tweag:lib-property-tests
Closed

lib: Create a generic property testing framework#212858
infinisil wants to merge 1 commit intoNixOS:masterfrom
tweag:lib-property-tests

Conversation

@infinisil
Copy link
Member

Description of changes

In #205190 some property tests were added, but extending those turns out to be pretty hard, which was noticed in #209099 when trying to add more.

This is a draft for a new generic property-testing framework that's pretty simple to use. It should eventually be usable for all the path library functions from #210426

This work is sponsored by Antithesis

Example

Here's a messed up regex for the lib.path.subpath.isValid check:

diff --git a/lib/path/default.nix b/lib/path/default.nix
index 96a9244407b..8c6fa15af44 100644
--- a/lib/path/default.nix
+++ b/lib/path/default.nix
@@ -34,7 +34,7 @@ let
     else if substring 0 1 value == "/" then
       "The given string \"${value}\" starts with a `/`, representing an absolute path"
     # We don't support ".." components, see ./path.md#parent-directory
-    else if match "(.*/)?\\.\\.(/.*)?" value != null then
+    else if match "(.*/)?\\.\\.(/.+)?" value != null then
       "The given string \"${value}\" contains a `..` component, which is not allowed in subpaths"
     else null;

And this is what you get when you try to run the property tests:

$ lib/path/tests/run.sh
Running property test normalise-append
Running property test normalise-error
Property test normalise-error failed: Expected variables valid and normalise_success to be the same, but they're not
[awk] subpath = gen_subpath() =
  ../
[awk] valid = subpath_is_valid(subpath) ? "true" : "false" =
  false
[nix] normalise_success = lib.boolToString (builtins.tryEval (lib.path.subpath.normalise subpath)).success =
  true
To reproduce run: lib/path/tests/run.sh normalise-error 15645

@infinisil
Copy link
Member Author

While this was interesting to create, and awk is a surprisingly good language (except the lack of local variables), the proper way to do this is to have stable language bindings for e.g. Python. For this there's already https://github.com/Mic92/pythonix, but that was archived due to requiring too much maintenance (since it relied on the unstable Nix library interface).

I updated pythonix to the latest Nix version in this fork's commit, but I'm considering upstreaming those bindings to Nix, such that they don't need to be maintained independently anymore.

@infinisil infinisil added the 1.severity: significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. label Apr 19, 2023
@roberth roberth added the 6.topic: lib The Nixpkgs function library label Jul 8, 2023
@infinisil
Copy link
Member Author

I'm closing this since there's now a PR to have Python bindings for Nix (outdated, won't get merged), and another one for C bindings (could get merged). Any of those approaches would be better than this hack here :).

@infinisil infinisil closed this Nov 9, 2023
@infinisil infinisil deleted the lib-property-tests branch November 9, 2023 00:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.severity: significant Novel ideas, large API changes, notable refactorings, issues with RFC potential, etc. 6.topic: lib The Nixpkgs function library

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants