[Python] Cython API header files seem to break editable installs

[AlenkaF]

Describe the bug, including details regarding any error messages, version, and platform.

I am running into a similar issue as discussed here: #49476 (macos user).

Import of pyarrow is successful from the arrow/ directory when doing a non-editable install (import is not working in arrow/python/ though). But trying to do an editable install doesn't work:

  *** Making editable...
  *** Created pyarrow-20.0.0.dev1549+g6dfef9428-cp314-cp314-macosx_26_0_arm64.whl
  Building editable for pyarrow (pyproject.toml) ... done
  Created wheel for pyarrow: filename=pyarrow-20.0.0.dev1549+g6dfef9428-cp314-cp314-macosx_26_0_arm64.whl size=4977549 sha256=ce413dfaa48170f292fdfec7209c96f572bcbfaf53ff231613961f75cb03584c
  Stored in directory: /private/var/folders/_h/78_6x1357y7f8nh15scyz69c0000gn/T/pip-ephem-wheel-cache-6ggs7dg9/wheels/43/1e/32/0b8e404ed4ea8d880029148f761ab6b9f028c7c3a2da5ca70f
Successfully built pyarrow
Installing collected packages: pyarrow
  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 20.0.0.dev1549+g6dfef9428
    Uninstalling pyarrow-20.0.0.dev1549+g6dfef9428:
      Created temporary directory: /private/var/folders/_h/78_6x1357y7f8nh15scyz69c0000gn/T/pip-uninstall-okf0ryow
      Removing file or directory /Users/alenkafrim/Repos/pyarrow-dev/lib/python3.14/site-packages/__pycache__/_pyarrow_editable.cpython-314.pyc
      Created temporary directory: /private/var/folders/_h/78_6x1357y7f8nh15scyz69c0000gn/T/pip-uninstall-hrn577ot
      Removing file or directory /Users/alenkafrim/Repos/pyarrow-dev/lib/python3.14/site-packages/_pyarrow_editable.pth
      Removing file or directory /Users/alenkafrim/Repos/pyarrow-dev/lib/python3.14/site-packages/_pyarrow_editable.py
      Created temporary directory: /Users/alenkafrim/Repos/pyarrow-dev/lib/python3.14/site-packages/~yarrow-20.0.0.dev1549+g6dfef9428.dist-info
      Removing file or directory /Users/alenkafrim/Repos/pyarrow-dev/lib/python3.14/site-packages/pyarrow-20.0.0.dev1549+g6dfef9428.dist-info/
      Created temporary directory: /Users/alenkafrim/Repos/pyarrow-dev/lib/python3.14/site-packages/~yarrow
      Removing file or directory /Users/alenkafrim/Repos/pyarrow-dev/lib/python3.14/site-packages/pyarrow/
      Successfully uninstalled pyarrow-20.0.0.dev1549+g6dfef9428

Successfully installed pyarrow-20.0.0.dev1549+g6dfef9428
Remote version of pip: 26.0.1
Local version of pip:  26.0.1 

and then trying to import:

(pyarrow-dev) ➜  python git:(main) ✗ python -c "import pyarrow"                                  
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import pyarrow
  File "/Users/alenkafrim/Repos/arrow/python/pyarrow/__init__.py", line 59, in <module>
    from pyarrow.lib import (BuildInfo, CppBuildInfo, RuntimeInfo, set_timezone_db_path,
    ...<3 lines>...
                             io_thread_count, is_opentelemetry_enabled, set_io_thread_count)
ModuleNotFoundError: No module named 'pyarrow.lib'

I investigated the issue with Claude and this diff fixed it:

diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt
index 6395b3e1e7..388e9be4f5 100644
--- a/python/CMakeLists.txt
+++ b/python/CMakeLists.txt
@@ -968,6 +968,9 @@ foreach(module ${CYTHON_EXTENSIONS})
         continue()
       endif()
     endif()
+    if(output MATCHES "\\.h$")
+      continue()
+    endif()
     install(FILES ${CMAKE_CURRENT_BINARY_DIR}/${output} DESTINATION ".")
   endforeach()
 endforeach()

The lib.h and lib_api.h were installed in the same dir as the compiled .so files causing issues when looking for lib.*.so compiled file. Note, lib.h and lib_api.h are already installed to arrow/python:

arrow/python/CMakeLists.txt

Lines 987 to 988 in d08d5e6

	install(FILES "${ARROW_PYTHON_BINARY_DIR}/lib_api.h" "${ARROW_PYTHON_BINARY_DIR}/lib.h"
	DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/arrow/python)

Component(s)

Python

[AlenkaF]

cc @raulcd

[raulcd]

Thanks @AlenkaF for the investigation.

I've just found this might also be failing on the nightly verification jobs:

• verify-rc-source-python-linux-almalinux-10-amd64
• verify-rc-source-python-linux-conda-latest-amd64
• verify-rc-source-python-linux-ubuntu-22.04-amd64
• verify-rc-source-python-linux-ubuntu-24.04-amd64
• verify-rc-source-python-macos-amd64
• verify-rc-source-python-macos-arm64
• verify-rc-source-python-macos-conda-amd64

those fail with:

   Created wheel for pyarrow: filename=pyarrow-24.0.0.dev231+gd08d5e64f-cp312-cp312-linux_x86_64.whl size=6825001 sha256=7951308367f77fa770fe247570a9160f67c8a60390ce4818eb9b9e696e4ce379
  Stored in directory: /tmp/pip-ephem-wheel-cache-rpeydwfw/wheels/f7/43/d8/264fa7f0cb8b0025fcf1961b8c821f346776a5c2b70364ba54
Successfully built pyarrow
Installing collected packages: pyarrow
Successfully installed pyarrow-24.0.0.dev231+gd08d5e64f
Traceback (most recent call last):
  File "<string>", line 2, in <module>
  File "/arrow/python/pyarrow/__init__.py", line 59, in <module>
    from pyarrow.lib import (BuildInfo, CppBuildInfo, RuntimeInfo, set_timezone_db_path,
ModuleNotFoundError: No module named 'pyarrow.lib'
Failed to verify release candidate. See /tmp/arrow-HEAD.LA9T8 for details.

[AlenkaF]

Yes, they do seem related. This is the same failed output I get locally:

(pyarrow-dev) ➜  python git:(main) ✗ python -c "import pyarrow"                       
Traceback (most recent call last):
  File "<string>", line 1, in <module>
    import pyarrow
  File "/Users/alenkafrim/Repos/arrow/python/pyarrow/__init__.py", line 59, in <module>
    from pyarrow.lib import (BuildInfo, CppBuildInfo, RuntimeInfo, set_timezone_db_path,
    ...<3 lines>...
                             io_thread_count, is_opentelemetry_enabled, set_io_thread_count)
ModuleNotFoundError: No module named 'pyarrow.lib'

I have opened a PR with the proposed patch. Waiting for the CI and the extended builds now.