CI: `cuda-pathfinder` is re-installed when running pip install (why?)

[leofang]

This

cuda-python/.github/workflows/test-wheel-linux.yml

Line 282 in 1e9e360

pip install --only-binary=:all: -v . --group "test-cu${TEST_CUDA_MAJOR}"

and this

cuda-python/.github/workflows/test-wheel-windows.yml

Line 255 in 1e9e360

pip install --only-binary=:all: -v . --group "test-cu${TEST_CUDA_MAJOR}"

re-install cuda-pathfinder from the public PyPI, instead of the one built and installed from the main branch (and used in the previous test steps). Example log:
https://github.com/NVIDIA/cuda-python/actions/runs/19456339177/job/55671231392#step:29:298

I don't think we should pass --only-binary=:all:. We probably need a more fine-grained installation command to exclude cuda-pathfinder.

[rwgk]

It does reinstall the version from the main branch in the example log you pointed out:

https://github.com/NVIDIA/cuda-python/actions/runs/19456339177/job/55671231392#step:29:310

  Attempting uninstall: cuda-pathfinder
    Found existing installation: cuda-pathfinder 1.3.3a0
    Uninstalling cuda-pathfinder-1.3.3a0:
      Removing file or directory /opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/cuda/pathfinder/
      Removing file or directory /opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/cuda_pathfinder-1.3.3a0.dist-info/
      Successfully uninstalled cuda-pathfinder-1.3.3a0

Successfully installed cuda-pathfinder-1.3.3a0 cutensor-cu12-2.3.1 nvidia-cublas-cu12-12.9.1.4 nvidia-cublasmp-cu12-0.6.0.84 nvidia-cuda-cccl-cu12-12.9.27 nvidia-cuda-nvcc-cu12-12.9.86 nvidia-cuda-nvrtc-cu12-12.9.86 nvidia-cudss-cu12-0.7.1.6 nvidia-cufft-cu12-11.4.1.4 nvidia-cufftmp-cu12-11.4.0.6.post1 nvidia-cufile-cu12-1.14.1.1 nvidia-curand-cu12-10.3.10.19 nvidia-cusolver-cu12-11.7.5.82 nvidia-cusparse-cu12-12.5.10.65 nvidia-cusparselt-cu12-0.8.1 nvidia-libmathdx-cu12-0.2.3.14.post1 nvidia-nccl-cu12-2.28.9 nvidia-npp-cu12-12.4.1.87 nvidia-nvfatbin-cu12-12.9.82 nvidia-nvjitlink-cu12-12.9.86 nvidia-nvjpeg-cu12-12.4.0.76 nvidia-nvshmem-cu12-3.4.5
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
+ pip list
WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Package                  Version        Build
------------------------ -------------- -----
cuda-bindings            12.9.4
cuda-core                0.4.2
cuda-pathfinder          1.3.3a0

It looks like it's actually working as intended? — I'd be surprising if the . in the pip install command did not mean "this right here"?

I looked systematically, pulling the logs from this run:

• https://github.com/NVIDIA/cuda-python/actions/runs/19451973661 (I picked this because it worked without reruns)

rwgk-win11.localdomain:~/logs_19451973661 $ grep -E -a 'Z +cuda-pathfinder +' *Test*.txt | grep -v 'cuda-pathfinder is already installed with the same version'
10_Test linux-aarch64 _ py3.12, 13.0.2, wheels, GPU a100.txt:2025-11-18T02:47:49.3926722Z cuda-pathfinder        1.3.3a0
11_Test linux-aarch64 _ py3.12, 12.9.1, local, GPU a100.txt:2025-11-18T02:54:07.3637275Z cuda-pathfinder          1.3.3a0
12_Test linux-aarch64 _ py3.13, 13.0.2, local, GPU a100.txt:2025-11-18T02:47:35.2485449Z cuda-pathfinder        1.3.3a0
13_Test linux-aarch64 _ py3.14, 13.0.2, local, GPU a100.txt:2025-11-18T02:47:41.6099639Z cuda-pathfinder        1.3.3a0
14_Test linux-aarch64 _ py3.11, 12.9.1, wheels, GPU a100.txt:2025-11-18T02:46:22.7039615Z cuda-pathfinder          1.3.3a0
15_Test linux-aarch64 _ py3.14t, 13.0.2, local, GPU a100.txt:2025-11-18T02:47:59.8393348Z cuda-pathfinder        1.3.3a0
16_Test linux-aarch64 _ py3.11, 13.0.2, local, GPU a100.txt:2025-11-18T02:47:25.9680294Z cuda-pathfinder        1.3.3a0
17_Test linux-aarch64 _ py3.10, 13.0.2, wheels, GPU a100.txt:2025-11-18T02:46:19.9658783Z cuda-pathfinder        1.3.3a0
18_Test linux-aarch64 _ py3.13, 12.9.1, wheels, GPU a100.txt:2025-11-18T02:46:21.0302809Z cuda-pathfinder          1.3.3a0
19_Test linux-aarch64 _ py3.10, 12.9.1, local, GPU a100.txt:2025-11-18T02:47:21.9414949Z cuda-pathfinder          1.3.3a0
1_Test win-64 _ py3.14t, 13.0.2, wheels, GPU t4.txt:2025-11-18T03:06:01.4449473Z cuda-pathfinder        1.3.3a0
21_Test linux-64 _ py3.13, 13.0.2, local, GPU H100.txt:2025-11-18T02:47:18.1149390Z cuda-pathfinder        1.3.3a0
22_Test linux-64 _ py3.14, 13.0.2, local, GPU l4.txt:2025-11-18T03:01:19.3499640Z cuda-pathfinder        1.3.3a0
23_Test linux-64 _ py3.10, 13.0.2, wheels, GPU l4.txt:2025-11-18T02:45:57.1521526Z cuda-pathfinder        1.3.3a0
24_Test linux-64 _ py3.14t, 13.0.2, local, GPU l4.txt:2025-11-18T02:46:48.4542641Z cuda-pathfinder        1.3.3a0
25_Test linux-64 _ py3.13, 12.9.1, wheels, GPU v100.txt:2025-11-18T02:46:03.0668339Z cuda-pathfinder          1.3.3a0
26_Test linux-64 _ py3.12, 13.0.2, wheels, GPU l4.txt:2025-11-18T02:45:40.4261298Z cuda-pathfinder        1.3.3a0
27_Test linux-64 _ py3.11, 12.9.1, wheels, GPU rtxpro6000.txt:2025-11-18T02:59:16.2687428Z cuda-pathfinder          1.3.3a0
28_Test linux-64 _ py3.10, 12.9.1, local, GPU v100.txt:2025-11-18T02:46:29.8429770Z cuda-pathfinder          1.3.3a0
29_Test linux-64 _ py3.13, 13.0.2, local, GPU rtxpro6000.txt:2025-11-18T03:04:00.2330786Z cuda-pathfinder        1.3.3a0
2_Test win-64 _ py3.14t, 13.0.2, local, GPU l4.txt:2025-11-18T02:56:16.1469314Z cuda-pathfinder        1.3.3a0
30_Test linux-64 _ py3.12, 12.9.1, local, GPU l4.txt:2025-11-18T02:50:54.5136113Z cuda-pathfinder          1.3.3a0
31_Test linux-64 _ py3.11, 13.0.2, local, GPU l4.txt:2025-11-18T02:46:47.6485111Z cuda-pathfinder        1.3.3a0
3_Test win-64 _ py3.12, 12.9.1, local, GPU t4.txt:2025-11-18T03:17:27.9228927Z cuda-pathfinder          1.3.3a0
4_Test win-64 _ py3.14, 13.0.2, wheels, GPU t4.txt:2025-11-18T03:04:32.6234390Z cuda-pathfinder        1.3.3a0
5_Test win-64 _ py3.13, 13.0.2, local, GPU l4.txt:2025-11-18T02:56:58.1160474Z cuda-pathfinder        1.3.3a0
6_Test win-64 _ py3.14, 13.0.2, local, GPU l4.txt:2025-11-18T02:56:25.3569183Z cuda-pathfinder        1.3.3a0
7_Test win-64 _ py3.13, 13.0.2, wheels, GPU t4.txt:2025-11-18T03:04:35.0383512Z cuda-pathfinder        1.3.3a0
8_Test win-64 _ py3.12, 12.9.1, wheels, GPU l4.txt:2025-11-18T02:54:48.4971593Z cuda-pathfinder          1.3.3a0

[leofang]

But this bit of log (which you also saw):

  Attempting uninstall: cuda-pathfinder
    Found existing installation: cuda-pathfinder 1.3.3a0
    Uninstalling cuda-pathfinder-1.3.3a0:
      Removing file or directory /opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/cuda/pathfinder/
      Removing file or directory /opt/hostedtoolcache/Python/3.10.19/x64/lib/python3.10/site-packages/cuda_pathfinder-1.3.3a0.dist-info/
      Successfully uninstalled cuda-pathfinder-1.3.3a0

Successfully installed cuda-pathfinder-1.3.3a0 (...)

is confusing to me. The CI by design installs things from bottom up, and so what's already installed (in this case, cuda-pathfinder) should not need to be installed again. I don't quite understand what is happening. Why would it be uninstalled, thus exposing us to the risk of re-installing it from a source that we are uncertain about? I don't recall seeing this bit of log in the beginning... 🤔

[rwgk]

I changed the title to reflect what's actually happening.

This is working as intended, it's just a little surprising at first:

• https://chatgpt.com/share/6935c566-1088-8008-aa3b-05399c906287

The re-installation overhead in this case is really minimal, compared to everything else in a CI run. I'll close this issue.

---

User Prompt

Please see attached script and log.

When running pip install twice, we see:

      Successfully uninstalled cuda-pathfinder-1.3.4a0
Successfully installed cuda-pathfinder-1.3.4a0
___

It seems intentional, but has been surprising in some situations.

Do you know about this behavior? Is the rationale documented somewhere?

Assistant Response

Yes — what you’re seeing is intentional pip behavior for local/editable installs, and it’s not just you. 🙂

From your log we can see:

• First call: pip install -v -e . --group test builds the editable wheel and installs cuda-pathfinder-1.3.4a0.
• Second call, same command, same version: pip still does

Attempting uninstall: cuda-pathfinder
  Found existing installation: cuda-pathfinder 1.3.4a0
  Uninstalling cuda-pathfinder-1.3.4a0:
    ...
Successfully installed cuda-pathfinder-1.3.4a0

even though the version didn’t change.

---

What’s going on?

There are three overlapping things here:

1. Local project installs are “special”

When you install from a path (like pip install . or pip install -e .), pip treats that differently from “normal” installs from an index:

• Local/editable installs trigger a rebuild every time.
• Pip always generates a new wheel (or editable wheel) and installs it.
• Therefore, pip sees an existing dist of the same version and performs an uninstall+install cycle.

This changed in pip 10: pip 9 used to say “Requirement already satisfied”, but pip 10+ reinstalls.

2. The rationale: local versions are untrustworthy

Pip maintainers have stated that local project versions cannot be trusted to represent identical contents:

• Developers often modify source without bumping versions.
• Re-running pip install . is the canonical workflow for picking up code or metadata changes.
• Therefore pip assumes reinstalling is safer than skipping.

Tools like uv replicate the same behavior intentionally.

3. Editable installs (PEP 660)

PEP 660 defines how editable installs work:

• Pip calls the backend’s build_editable hook.
• The produced wheel is ephemeral and must not be cached.
• Pip installs it like any other wheel, triggering uninstall+install when the dist name/version matches.

Thus pip install -e . by definition reruns build + install every time.

---

Is this behavior documented?

The rationale is spread across:

• Pip’s internal docs (editable installs are a “special case”).
• PEP 660 (editable wheels are ephemeral).
• Pip issue discussions (pip 10+ always reinstalls local packages).
• Downstream tool discussions (e.g., uv adopting the same behavior).

There is no single documentation page with a clear headline saying “pip always reinstalls local projects”, but the behavior is consistently intentional.

---

Practical TL;DR

• Running pip install . or pip install -e . always reinstalls the package.
• Pip assumes you are developing locally and may have changed code without updating the version.
• This is why you see Successfully uninstalled ... followed by Successfully installed ... even on the second run.