Running in a secondary virtualenv fails

[mattip]

Try this:

• create a venv and install pkg-config into it, then run pkgconf, all is good.

  $ python3.13 -m venv /tmp/venv_with_pkg_config
  $ source /tmp/venv_with_pkg_config/bin/activate
  $ pip install pkgconf
  $ deactivate
  $ /tmp/venv_with_pkg_config/bin/pkgconf --version
  # prints 2.5.1 and debug stuff to stderr
  

• Now create another venv, and try to call pkgconf with an absolute path. It fails.

  $ python3.13 -m venv /tmp/venv2
  $ source /tmp/venv2/bin/activate
  $ /tmp/venv_with_pkg_config/bin/pkgconf --version
  # it errors!!!
  

• Unset $VIRTUAL_ENV, it will run again

  $ unset VIRTUAL_ENV
  $ /tmp/venv_with_pkg_config/bin/pkgconf --version
  # prints 2.5.1 and debug stuff to stderr
  

This is intentional.
However it breaks using the package as a true replacement for pkg-config, since the utility is meant to be called inside an isolated virtual environment in pip install or cibuildwheel. See numpy/numpy#26574.

[mattip]

Can we get a IGNORE_VIRTUAL_ENV environment variable to override this code and use the shebang as-provided?

[virtuald]

IMHO pkgconf should only detect entrypoints in the environment that it was installed in.

It seems like the code was added in #59 to allow --system-site-packages to work? But I'm surprised that the re-execution would be necessary for that.

[mattip]

There are two opposing scenarios. The one in this issue is laid out above.

The other scenario, fixed in #59, is that somehow people install pkgconf into the system python, then create a virtualenv with --system-site-packages, then install pkgconf. Now which pkgconf did they intend to use? I think the goal of #59 is to make sure they get the one installed into the virtualenv. I will note that pip makes it hard to install packages into the system python. On debian it will error, and on rehat/Fedora warn with

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.

I guess the questions are what should be the default, how the choice can be overridden, and how can the situation be described to the unaware package user.

For reporting, it would be nice to add to the stderr logging that the code in #59 was activated. This would have saved me a couple of hours of debugging what was going on in numpy.

For default/overriding: I suggest either

• an additional environment variable to allow skipping the code added in py: fix entrypoints in --system-site-packages venvs #59, or
• somehow detecting the situation described in py: fix entrypoints in --system-site-packages venvs #59 and rather than blindly running a different pkgconf, try to detect that other pkgconf and error out if it is found. If it is not found, continue to run the current pkgconf.

[virtuald]

#98 is mildly related.

[FFY00]

Let me just clarify something, you trying to use pkgconf-pypi to find system dependencies in this use-case, are you? Because that is fundamentally incompatible.

AFAICT, the other option would be that you are trying to use a dependency from another virtual environment. In this case, why?
We can make it work by running on both environments, but I am struggling to see why that would be needed.

That said, the lack of debugging information to help identify what is happening is indeed very unfortunate. Sorry!

For reporting, it would be nice to add to the stderr logging that the code in #59 was activated. This would have saved me a couple of hours of debugging what was going on in numpy.

I agree 👍

The other scenario, fixed in #59, is that somehow people install pkgconf into the system python, then create a virtualenv with --system-site-packages, then install pkgconf. Now which pkgconf did they intend to use? I think the goal of #59 is to make sure they get the one installed into the virtualenv.

The user doesn't need to install pkgconf in the venv. The issue is that if it was installed in the system Python, the pkgconf entrypoint script will run under /usr/bin/python, not .../venv/bin/python, so the venv packages won't be in sys.path. This means the script will fail to find packages from the venv that provide a pkg_config entrypoint.

It seems like the code was added in #59 to allow --system-site-packages to work? But I'm surprised that the re-execution would be necessary for that.

Re-execution is needed here because there may be some dynamic setup of sys.path or sys.meta_path tied to the environment startup, via a ._pth file, or the site module (.pth files, or sitecustomize; not sure if usercustomize applies here).

[FFY00]

AFAICT, the other option would be that you are trying to use a dependency from another virtual environment. In this case, why?
We can make it work by running on both environments, but I am struggling to see why that would be needed.

In fact, this would lead to #55.

[mattip]

Let me just clarify something, you trying to use pkgconf-pypi to find system dependencies in this use-case, are you? Because that is fundamentally incompatible.

Yes, I understand it is currently incompatible. But why "fundamentally"? It could be made compatible, making this repo useful for more people. Is there some a reason you don't want to support it as an alternative, and that providing a command line option or environment variable to support #55 is a big problem for the package?

you are trying to use a dependency from another virtual environment. In this case, why?

Yes, this is exactly the use case. See numpy/numpy#26574. We don't want to make scipy-openblas wheels a hard build dependency of numpy, rather make it an optional build dependency. The requirements for this are a bit problematic, especially when considering building an sdist tarball. The way we worked out to do it (without pkgconf-pypi) is to create a virtualenv, add the scipy-openblas wheel there, then set PKG_CONFIG_PATH and run the build. This all works, if there is a pkg-config available. But windows does not make it easy to get pkg-config, (and the one available does not support $prefix), we have to donwload it from sourceforge in CI making the whole process harder. This package goes one better: we can (almost) use pkgconf-pypi installed in the virtualenv together with scipy-openblas without needing PKG_CONFIG_PATH. But that use case is blocked with no option to override it.

[virtuald]

You could just get scipy-openblas register the entrypoint that pkgconf-pypi uses, and then it would be able to find it?

[FFY00]

Yes, I understand it is currently incompatible. But why "fundamentally"? It could be made compatible, making this repo useful for more people. Is there some a reason you don't want to support it as an alternative, and that providing a command line option or environment variable to support #55 is a big problem for the package?

Because we know nothing about the system — we are not the system administrators. Only them know how pkg-config/pkgconf should be configured to correctly resolve dependencies on the system. We could try common prefixes, and while that would probably work on a number of POSIX systems, on something like NixOS, where every package is namespaced, it would be impossible for us to ever get it working. Even on those POSIX systems, we might end up resolving dependencies differently than what was expected. This is why I say that use-case is fundamentally incompatible with what we can provide.

That's the reason we have gone through all this trouble to get the semi-wrap in the first place — because we know how to resolve pkg-config dependencies on the Python world, but we must yield to the system on system dependencies 🫤. If we could resolve system dependencies directly, we could just do that instead having a semi-wrap.

[FFY00]

Looking at the linked PR, it looks like one of the main issues is our semi-wrap failing to run when pkgconf isn't installed on the second venv, which I think could be a reasonable use-case — you want to us to resolve a Python dependency in a build environment we are not installed.

[FFY00]

Anywhow, this limitation should probably be documented!

[mattip]

Any more thoughts how to progress here?

[virtuald]

I haven't given it a ton more thought.

I'm still really attracted my proposal in #98, having an executable pkgconf just be vanilla pkgconf with no enhancements and pkgconf-pypi be juiced with our special entrypoint resolving stuff. Then it's sorta like cmake and ninja and other binaries that you install via pip -- they just work the way you might expect them to.

The key prereq would be to make meson-python/hatch-meson be able to easily insert an override so that it uses the pkgconf-pypi executable instead of pkgconf when configured to do so. This should be straightforward, I just haven't taken the time to do it.

[FFY00]

Alright, let's do that then. I just have a couple UX details I think we should do:

• Since the pkgconf binary we ship cannot ever work without setting PKG_CONFIG_PATH, let's emit a warning if it is called without it set (this also helps existing users with the migration to the new model)
• For cases when tools don't let us provide a custom name for the pkgconf binary, let's add a environment flag to enable the pkgconf-pypi behavior in pkgconf

If you really don't want to wrap the binary, which we would need to keep doing for the functionality above to work, we can try to ask the upstream to support building it without a target prefix, and letting them emit the warning or fail if PKG_CONFIG_PATH isn't set. If this is not accepted, we can also split this into two packages — a vanilla pkgconf, and pkgconf-pypi.

Let me know. I can implement the changes, we just need to decide on this.