Name mappings: questions wrt `version_operators`

[mgorny]

I don't want to derail #44, so here's a way things I've noticed while looking at it:

1. The spec doesn't really seem to say what we do with version_operators. Perhaps including some versions in the command call examples would help.
2. It is said that "Empty strings can be used as a value if that particular operator is not supported." What should happen then? Makes me wonder if we should try to reduce the number of supported operators, since Python version syntax is quite baroque compared to other package managers, and this may lead to things like =~ being dropped while roughly equivalent >=,< would work.
3. I'm guessing this assumes infix operators. Gentoo uses prefix operators, i.e. {op}{package}-{version}. Might be fixable by using templates rather than just operators.
4. Do we allow multiple version specifiers per package? Again, these might not be trivially mapped to other ecosystems. In Gentoo, you can only use one operator per package dependency. You can technically do ranges by specifying multiple dependency specifiers (and AND-ing result of multiple query commands), but this won't yield correct results when packages are slotted (i.e. permit multiple versions being installed simultaneously).

[jaimergp]

Thanks for the feedback! I'll reply here and then we can open a PR to implement what we agree on.

The spec doesn't really seem to say what we do with version_operators. Perhaps including some versions in the command call examples would help.

So far I have only implemented it for the conda package managers:

      "version_operators": {
        "and": ",",
        "arbitrary": "==",
        "compatible": "~=",
        "equal": "=",
        "greater_than": ">",
        "greater_than_equal": ">=",
        "less_than": "<",
        "less_than_equal": "<=",
        "not_equal": "!=",
        "separator": ""
      }

Meaning that a PyPI spec like dep:generic/package@2.0 becomes package==2.0, while dep:generic/package@>=2.0 becomes package>=2.

It is said that "Empty strings can be used as a value if that particular operator is not supported." What should happen then? Makes me wonder if we should try to reduce the number of supported operators, since Python version syntax is quite baroque compared to other package managers, and this may lead to things like =~ being dropped while roughly equivalent >=,< would work.

Yes, I see your point and I like it. Maybe we can get away with and, equal, greather_than_equal, less_than and not_equal. But we need to restrict that in PEP 725 first. Once we agree on what we want to support in the dep: syntax, we can restrict this map.

I'm guessing this assumes infix operators. Gentoo uses prefix operators, i.e. {op}{package}-{version}. Might be fixable by using templates rather than just operators.

Yes, your assumption is correct. Using a template also allows us to drop separator, so this makes sense.

Do we allow multiple version specifiers per package?

Yes, with and (PEP440's ,). In its absence, we can "explode it" into several single-constraint specs. What do you think?

[mgorny]

Maybe we can get away with and, equal, greather_than_equal, less_than and not_equal. But we need to restrict that in PEP 725 first. Once we agree on what we want to support in the dep: syntax, we can restrict this map.

I think I'd go for the full set of >=, >, <, <= there, for symmetry. Other than that, LGTM.

Yes, with and (PEP440's ,). In its absence, we can "explode it" into several single-constraint specs. What do you think?

Yeah, I think that's the best we can do.

[jaimergp]

Well, I surveyed all the ecosystems we are currently demoing and the situation is a bit more complex:

• arch: No version selection supported.
• chocolatey: choco install {package_name} --version={version}. One package at a time if a version needs to be specified.
• conan: conan install {package_name}/{version}, conan install {package_name}/[>={version} <{version}].
• conda-forge: conda install {package_name}=<version, conda install {package_name}<={version},>{version}
• fedora: No version selection supported.
• gentoo: emerge ={package_name}-{version}, emerge >={package_name}-{version} <{package_name}-{version}.
• homebrew: brew install postgresql@15, only with some formulaes.
• nix: No version selection supported.
• pypi: pip install <package>=={version}, ranges supported.
• scoop: scoop install {package_name}@{version}.
• spack: spack install {package_name}@={version}, spack install {package_name}@{version}:{version}.
• ubuntu: No version selection supported.
• vcpkg: vcpkg install {package_name}@{version}.
• winget: winget install {package_name} --version {version}. One package at a time if a version needs to be specified.

Things I learned:

• Not all ecosystems support version specification at all.
• Not all package managers support specifying version ranges (only a few!).
• Not all packages use infixes for version specification. Some use a separate argument.
• All package managers allow several package names at once, but some will only accept one at a time if the version needs to be specified.

This creates the following requirements:

• Different levels of support: no versioning, only exact versions, version ranges. They might have different syntaxes.
• The version specification syntax may need several arguments per package name.
• Specifying a version might switch the package manager into a single-spec tool.

I will try to come up with a solution for this, but it's getting tricky.

[mgorny]

We probably need to distinguish between "version accepted for install" and "version accepted for queries" too. In the end, if the PM doesn't accept version for install but we require a newer one, I suppose asking the user to try upgrading may work too.

Or we may also want to query available packages, but I think that'd be an overkill.

[jaimergp]

Most package managers do not allow specifying a version constraint in the query anyway, so I'm inclined to only allow names there.

[jaimergp]

This is what I'm coming up with:

{
    "specifier_syntax": {
        "multiple_specifiers": "always", // "name-only", "never"
        "exact_version": ["{name}@{version}"], // ["{name}", "--version", "{version}"], false
        "version_ranges": {
            "syntax": ["{name}{range}"], // ["{op}{name}-{version}"], ["{name}/[{range}]"]
            "range": "{op}{version}", // "{op}{name}-{version}"
            "and": ",", // if false, explode into several specs
            "equal": "=",
            ...
        }, // can also be false for not supported
    }
}

For arch, where no version selection is allowed:

{
    "specifier_syntax": {
        "multiple_specifiers": "name-only",
        "exact_version": false,
        "version_ranges": false
    }
}

For scoop, only exact versions:

{
    "specifier_syntax": {
        "multiple_specifiers": "always",
        "exact_version": ["{name}@{version}"],
        "version_ranges": false
    }
}

For winget, only exact versions:

{
    "specifier_syntax": {
        "multiple_specifiers": "name-only",
        "exact_version": ["{name}", "--version", "{version}"],
        "version_ranges": false
    }
}

For conda, with full support for version ranges:

{
    "specifier_syntax": {
        "multiple_specifiers": "always",
        "exact_version": ["{name}=={version}"],
        "version_ranges": {
            "syntax": ["{name}{range}"],
            "range": "{op}{version}", // if multiple ranges are present, they are "{and}".join()'d 
            "and": ",",
            "equal": "=",
            ... all operators mapped here
        }
    }
}

For gentoo, with full support for version ranges, but exploded in multiple specs:

{
    "specifier_syntax": {
        "multiple_specifiers": "always",
        "exact_version": ["{name}-{version}"],
        "version_ranges": {
            "syntax": ["{op}{name}-{version}"],
            "range": null,
            "and": false, // explode into multiple specs using 'syntax'
            "equal": "=",
            ... all operators mapped here
        }
    }
}

For conan, with support for version ranges but using different syntax than exact version:

{
    "specifier_syntax": {
        "multiple_specifiers": "always",
        "exact_version": ["{name}/{version}"],
        "version_ranges": {
            "syntax": ["{name}/[{range}]"],
            "range": "{op}{version}",
            "and": " ",
            "equal": "=",
            ... all operators mapped here
        }
    }
}

[mgorny]

Most package managers do not allow specifying a version constraint in the query anyway, so I'm inclined to only allow names there.

Oh my, I'm surprised that they don't have a function dedicated to machine querying.

---

I think that generally looks good. I'm slightly worried it's a bit ad-hoc, but perhaps it's good enough to cover foreseeable cases. One thing I'd change is replace false with nulls everywhere.

[rgommers]

This looks quite reasonable to me. Yeah it's a little complex, but we live in a messy world. I'd be happy to go with this (and don't have any simpler ideas), the worst that can happen is that there'll be a request to take it out of the PEP.

[jaimergp]

Yea, the only other idea I had is to define a CLI interface that implementors would have to create for each package manager, delegating the whole effort to the ecosystem maintainers.

I'm going to give this schema a try and implement it, and we'll see how it looks like in code. Thanks for the feedback!

[eli-schwartz]

These claims sound bogus. I'd expect e.g. that any package manager that supports a query syntax would allow the same values that can be passed as installation requests on the command line, and being able to request version specifiers to install is pretty important functionality for example to provide a documented command to upgrade a package as part of a security bulletin:

foobarpkg in version 1.2.3 has a critical security 
vulnerability, fixed in 1.2.4.

Resolution:

Upgrade to the newest packaged release:

# pacman -Syu 'foobarpkg>=1.2.4'

Yes, that's explicitly supported by:

arch: No version selection supported

And is roughly the language used in security advisories as well.

You can do pacman -T 'foobarpkg>=1.2.4' to test (-T is --deptest) if the package is already installed with a satisfactory version rather than unconditionally updating.

And,

ubuntu: No version selection supported.

On Debian, apt install takes foobarpkg=1.2.4 but cannot do version constraints so you must know exact versions. However apt satisfy 'foobarpkg (>= 1.2.4)' exists and supports the style used in dependency fields for a package, which obviously supports a range of comparison types. The reason for two subcommands is unknown to me but at a guess it's because legacy compat and because random end users are less likely to be familiar with the somewhat complicated syntax of a Debian control file.

...

I'm out of time right now so I'm not going to continue my analysis, but that's two misdiagnosed cases in the list from above and it's the first two that I checked (thouth I admit I cheated because I already knew the answer for pacman -- upstream maintainer prerogative. I had to look up Debian/Ubuntu).

I will say that I'm inclined to believe it would be foolish for any package manager¹ to not explicitly support this and I am skeptical of claims that they lack this feature and will not believe they lack it until I've seen it for myself. If they do genuinely miss the functionality then it is a P1 critical UX failure so please submit bug reports to all such package managers asking for the functionality. This is not just something the python ecosystem needs.

Footnotes

1. note I only really believe in distribution package managers so for example homebrew does NOT matter to me, but Fedora and nix do... ↩

[jaimergp]

Thanks for the feedback! I'll review the distros one by one, I mostly based my analysis using their online help, not interactive testing.

Yes, that's explicitly supported by:

There are no mentions of version constraints in https://wiki.archlinux.org/title/Pacman. The man pages do mention that, but it's unclear to me whether that's a pacman feature and not an "Arch Linux" one. IIRC (it's been a while since I used it), Arch Linux only supported upgrades, not downgrades. Also targeted version selections were discouraged. But maybe things have changed in these ~7 years.

---

About apt, my understanding is that Ubuntu only allows updates within their security fixes, so you can't really choose, you can just "upgrade" to those. How does satisfy work if there's only a version available at a time per distro release? Will it downgrade if necessary? I guess I need to run some tests.

---

Regardless, these two corrections (thanks again!) only mean that I need to update the specifier_syntax instructions in jaimergp/external-metadata-mappings#22. Unfortunately, there are still cases where no version constraints are allowed outside of the Linux world, so this convoluted schema is still needed.

[jaimergp]

Also, to clarigy, the full meaning behind "No version selection supported" I meant "this ecosystem doesn't allow users to install arbitrary versions". Mostly because they are Linux distributions that choose which package version (with backports) to ship in each release (Fedora, Ubuntu), or are of rolling nature and only the latest one is supported (Arch).

e.g. I can't install numpy=2 in Ubuntu Noble because only 1.26.4 is available there. I would need to dist-upgrade to Plucky to get 2.2.3.

Hence why it makes no sense to offer version selection in those ecosystems. What's not so clear to me is what do in those cases: drop the version constraint and see if we are lucky, error out? I think the idea of querying what's available and checking whether the range satisfies it is out of scope for the PEP and tools could choose to do that on their own, so I think it'll be left as an implementation detail.

[mgorny]

Well, yes and no. I think that for a general case, you may have a dep like foo >= 1.2.3, and having foo-devel-1.2.2 installed may imply that there's the possibility of upgrading to foo-1.2.3 or newer that would satisfy the constraint.

Of course, in the long run there's no guarantee that the upgrade will help, or that you'll be able to converge all dependencies on matching versions, but I guess it's better than nothing.

You'd probably still need appropriate operator syntax for querying packages, if query commands support specifying versions.