Language bindings and/or C interface for the Nix language

[roberth]

Is your feature request related to a problem? Please describe.

Some applications' needs aren't fully met by the CLI and its --json flags.
Examples:

• evaluation involving multiple function applications
• applications that have intricate data flows back and forth between Nix and the app
• applications that want to stream evaluation info
• static binaries for single file distribution

Developing and maintaining programs that use the C++ code requires more effort over time, because the C++ api is not stable.
Language bindings that are developed as an integral part of Nix reduce the total maintenance effort by removing the need for multi-version compatibility. Committing to support more interfaces may increase the burden on the team somewhat, except for the Haskell bindings, which are already maintained by a team member, me.

Describe the solution you'd like

We could take one of at least two directions, or keep the status quo.

1. Create a stable C facade, and build language bindings on top of that
2. Create bindings that interoperate directly with C++

Benefits of going through C

• broader support than C++
• if stable, linking against various versions of Nix is easy

Benefits of talking directly to C++

• performance may be better, especially if the C facade doesn't follow the contours of the C++ interface very closely. String conversions may be inefficient
• talking directly to C++ may be less effort, especially initially as Nix could absorb the hercules-ci-cnix-* Haskell bindings and pythonix

Both approaches are valid simultaneously, considering that a C facade and C bindings are basically the same thing. (Won't be a completely standard C library though, because of libstore's coroutines, libexpr's gc, and libexpr's stack overflow handler)

Describe alternatives you've considered

Status quo:

• Maintenance overhead is higher, potentially leading to more abandoned bindings (like pythonix).
• NixOps will keep suffering from missing data when going back and forth between resources and machine configs in certain ways.
• Executable language spec idea / doctester would have to be implemented in C++.

Additional context

• https://github.com/Mic92/pythonix
• https://www.tweag.io/blog/2022-09-01-unit-test-your-nix-code/
• https://hackage.haskell.org/packages/search?terms=hercules-ci-cnix

[infinisil]

Note that Nix also already has (I believe stable) Perl bindings.

A slight problem with more such bindings is that it increases the build-time closure, which would be especially problematic with Haskell. This can be worked around though by not building the bindings as part of the main build, but rather just copying the source code to the result. And to make sure they work and are maintained we can still have an CI check that does the building.

For improved testing in nixpkgs I'm now considering updating and upstreaming pythonix. I already managed to just update pythonix, but actually upstreaming is much harder.

So before I jump in too deep:

• Is pythonix something that would be accepted upstream?
  The interface of pythonix is pretty simple and limited, but I think it's a good start and is enough for my use case of testing evaluation:
  Just an eval function that takes a Nix expression as a string, a set of variable assignments (Python values which get converted to Nix values automatically), and the result which is converted to a Python value automatically.
• Any tips for how to get started? I'm neither a C++, autotools or python expert, so I'd appreciate some pointers.

Ping @NixOS/nix-team

[roberth]

A slight problem with more such bindings is that it increases the build-time closure

Only if it's part of the same derivation. The bindings should be in separate derivations.

Is pythonix something that would be accepted upstream?

+1 for me.
(and upstream is here :) )

• Any tips for how to get started? I'm neither a C++, autotools or python expert,

By putting the bindings in a separate derivation it can work the same as it does today.

I'd expect at least

• rename pythonix -> nix, as it is the official binding. The nix name seems to be available (PyPI)
• make sure it is well tested and documented
• clarify that the repository has both LGPL and MIT licensed code

Also fyi, we don't have source filtering yet, so you might want to carve out the python bindings directory while you're working in it. We'll filter the source of the main package after #6538.

[infinisil]

The bindings should be in separate derivations

Oh yeah that sounds pretty good

• The nix name seems to be available (PyPI)

I don't think adding it to Pypi is needed for a start. I expect most people wanting to use this library to also have their build be done in Nix, in which case they can just use the derivation directly.

[roberth]

While it's not strictly necessary, the package will be put into a namespace with the PyPI packages, whether through Nixpkgs or poetry2nix, so it's a good idea to at least claim the name.

[infinisil]

Started upstreaming Pythonix here: #7735

[Ericson2314]

Yeah I think this is very import. This helps us onboard more developers, and support more downstream projects; basically this is a great example of the "force multiplier" track that @fricklerhandwerk has talked about.

Having Guile Scheme libstore-only bindings would help with the NixOS/rfcs#134 future work I hope to get to, too.

[fricklerhandwerk]

Related: #8699