Integrate technical reference documentation from native_lib_loader

doc/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?= -n -v -W --keep-going
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

doc/make.bat

@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

doc/requirements.txt

@@ -0,0 +1,4 @@
+sphinx==8.2.3
+furo==2024.8.6
+numpydoc==1.8.0
+myst-parser==4.0.1

doc/source/conf.py

@@ -0,0 +1,50 @@
+"""Configuration file for the Sphinx documentation builder.
+
+For the full list of built-in configuration values, see the documentation:
+https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+-- Project information -----------------------------------------------------
+https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+"""
+
+# Basic project configuration
+project = "dynamic-library"
+copyright = "2025, Filipe Laíns"  # noqa: A001
+author = "Filipe Laíns"
+
+# -- General configuration ---------------------------------------------------
+extensions = [
+    "sphinx.ext.autodoc",  # Auto-generate API documentation
+    "sphinx.ext.viewcode",  # Add links to source code
+    "sphinx.ext.intersphinx",  # Link to other Sphinx documentation
+    "sphinx.ext.coverage",  # Check documentation coverage
+    "sphinx.ext.autosummary",  # Generate summary tables
+    "sphinx.ext.napoleon",  # Support for math equations
+    "numpydoc",  # Enhanced NumPy documentation support
+    "myst_parser",  # For parsing Markdown files
+]
+html_theme = "furo"
+templates_path = ["_templates"]
+
+# autodoc
+autodoc_typehints = "description"  # Put type hints in the description
+autodoc_member_order = "bysource"  # Order members by source order
+autodoc_default_options = {
+    "members": True,
+}
+autoclass_content = "class"  # Include both class and module docstrings
+
+# autosummary
+autosummary_generate = True  # Generate stub pages for API
+
+# intersphinx
+intersphinx_mapping = {
+    "python": ("https://docs.python.org/3", None),
+    "numpy": ("https://numpy.org/doc/stable", None),
+}
+
+# numpydoc
+numpydoc_class_members_toctree = False
+
+# Myst
+myst_heading_anchors = 4

doc/source/index.rst

@@ -0,0 +1,22 @@
+Managing Native Libraries in Python Wheels
+==========================================
+
+A major driving force in Python's explosive popularity is the ease with which it can be integrated with libraries written in lower-level compiled languages.
+These `extension modules <https://docs.python.org/3/extending/extending.html>`__ are distributed as shared libraries that are loaded by the Python interpreter at runtime.
+While extension modules are a powerful tool for Python developers, they are a common source of frustration when distributing Python packages.
+One of the most challenging problems associated with extension modules is how to manage the shared libraries that these extension modules depend on.
+
+Python's package manager, `pip <https://pip.pypa.io/en/stable/>`__, is primarily designed to install pure Python packages, and it struggles with the additional complexities required to produce safe, self-consistent software environments once complex extension modules with complex dependency trees are involved.
+Most of these challenges are well documented at `pypackaging-native <https://pypackaging-native.github.io/>`__, so this document will not attempt to duplicate that information.
+
+This package provides a hook-based mechanism to automatically load package-provided dynamic libraries at Python startup, enabling safe sharing of native libraries between Python wheels.
+The approach leverages Python's entry point system and ``.pth`` files to preload shared libraries before extension modules are imported, ensuring that dependencies are available when needed.
+
+The documentation provided here focuses on the underlying mechanics of how shared library loading works across different platforms and the various approaches that can be used to solve the library distribution problem.
+This technical background is essential for understanding why certain implementation choices were made and how to work effectively with native libraries in Python packages.
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   reference/index

doc/source/reference/background.md

@@ -0,0 +1,93 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES.
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Background
+
+## Motivation
+
+For the purpose of this document, we will distinguish between two types of languages: **[compiled languages](https://en.wikipedia.org/wiki/Compiled_language)** and **[interpreted languages](https://en.wikipedia.org/wiki/Interpreted_language)**.
+Code written in compiled languages must be translated into machine code before it can run, whereas interpreted code is run directly by an **interpreter** (usually itself a compiled program) at runtime.
+Python is an example of an interpreted language, while C++ is an example of a compiled language.
+For the rest of this document, we will use these two as representative examples of interpreted and compiled languages.
+
+Since Python is an interpreted language, why do we care about how compiled languages work?
+While there are multiple implementations of the Python interpreter, the most popular by far is [CPython](https://en.wikipedia.org/wiki/CPython), the _de facto_ reference implementation of Python written in C.
+A huge part of Python's success is due to its ability to leverage existing C libraries by writing [Python extension modules](https://docs.python.org/3/extending/extending.html) that leverage CPython's [C API](https://docs.python.org/3/c-api/index.html).
+This functionality is critical to Python's success, but it also in practice means that as a near-universal "glue language" Python libraries are effectively exposed to the set of library and packaging problems from every language that Python extension modules interact with.
+As such, the distribution of many of the most popular Python packages requires a deep knowledge of these pieces.
+
+## Libraries and Interfaces
+
+Ideally, code to achieve a particular task should be written once and reused.
+The typical unit of code sharing is a **[library](https://en.wikipedia.org/wiki/Library_(computing))**, which is a collection of useful bits of code (functions, classes, etc) that can be reused by other code.
+To be shareable, libraries must establish a contract, the [interface](https://en.wikipedia.org/wiki/Interface_(computing)), for how other code can interact with them.
+When a developer manages all the code in a project, the stability of the interface is not a major concern since all code can be updated together.
+When code is shared between different projects, however, the interface becomes a contract that must remain stable or risk breaking other code.
+
+### Programming and Binary Interfaces
+
+There are two primary types of interfaces relevant in to this discussion: the **[application programming interface (API)](https://en.wikipedia.org/wiki/Application_programming_interface)** and the **[application binary interface (ABI)](https://en.wikipedia.org/wiki/Application_binary_interface)**.
+The API is the set of objects (functions, classes, etc) that a library exposes publicly.
+For code written in interpreted languages, the API is the only relevant interface.
+For compiled code, the ABI is a second, lower-level interface that describes how code interacts with other code after it is compiled to machine instructions.
+The ABI is typically a stricter contract than the API because it includes rules about how data is passed between functions, how memory is managed, and other low-level details.
+While the API is entirely governed by source code, the ABI is additionally influenced by many other factors such as the compiler used or the operating system on which the code is run.
+
+## Linking and Loading
+
+For interpreted languages, libraries are nothing more than source code.
+For compiled languages, however, libraries are typically distributed in compiled form.
+To understand how compiled libraries are used, we must understand just a little bit of how [compilers](https://en.wikipedia.org/wiki/Compiler) work.
+Very crudely, compilation involves two steps:
+1. The translation into machine code. This step itself usually has many other sub-steps (preprocessing, translation into IR, assembling, etc) that we will not go into here.
+2. **[Linking](https://en.wikipedia.org/wiki/Linker_(computing))**: the machine code is combined with other machine code to form a single binary.
+
+Compiled libraries are distributed in one of two forms:
+1. [Static libraries](https://en.wikipedia.org/wiki/Static_library) are essentially archives of compiled code. When a project depending on a static library is compiled, the linking step essentially copies all relevant components of the static library into the binary. This produces binaries that are inflated in size but are relatively portable due to the lack of external dependencies.
+2. [Dynamic (aka shared) libraries](https://en.wikipedia.org/wiki/Dynamic-link_library) are loaded at runtime by the operating system.
+
+There are two ways in which dynamic libraries can be used, and these in turn depend on the [loader](https://en.wikipedia.org/wiki/Loader_(computing)) used by the operating system:
+1. Load-time dynamic linking: this is when a project declares a dependency on a dynamic library while being compiled. This is the more common way in which dynamic libraries are used because it allows the functions/classes/etc from the dependency to be used directly in code. For example, if I have a C header `foo.h` declaring a function `f(x)` that is defined in `foo.c`, `bar.c` would contain `#include "foo.h"` and then directly use `f(x)` in its code.
+2. Runtime dynamic linking: with this approach, the library is loaded using specific functions for library loading when the program executes (`dlopen` on [Linux](https://www.man7.org/linux/man-pages/man3/dlmopen.3.html) or [Mac](https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man3/dlopen.3.html)  and [LoadLibrary on Windows](https://learn.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-loadlibrarya)). Notably, runtime loading of a library also triggers transitive loading of all of _its_ dependencies. Runtime loading has the advantage that the main compiled program does not express a library dependency in its binary, so it can be loaded on a system where the dependent library does not exist. This approach is necessary when a library is intended to be optional, for instance, and is a standard practice for enabling a plugin architecture. This flexibility comes at a cost, however, because errors are observed at runtime instead of at link-time and can make code harder to debug.
+
+With all the above in mind, we are now prepared to understand how all of this information relates to Python extension modules.
+Python's C API can be used to embed calls to functions across various library interfaces using [foreign function interfaces](https://en.wikipedia.org/wiki/Foreign_function_interface) (FFIs).
+Python extension modules are nothing more than shared libraries (`.so` files on Linux, `.dylib` on OS X, and [`.pyd` on Windows](https://docs.python.org/3/faq/windows.html#is-a-pyd-file-the-same-as-a-dll)) that the Python interpreter loads as plugins via runtime dynamic linking when code tries to `import` them (for pure Python modules, `import` simply runs the Python source code in the package directly).
+
+## Packages and Package Managers
+
+**Packages** are a distribution mechanism for libraries.
+Packages bundle libraries with metadata that can be used to identify the interface of the bundled library.
+As libraries evolve, the metadata is updated to reflect the changes to the interface.
+**Wheels** are the standard package distribution format for Python packages.
+
+**Package managers** manage installation and removal of packages.
+Package managers enable creating consistent, reproducible environments of mutually compatible software.
+At a high level, package managers leverage package metadata to ensure that environments are kept in a valid state with compatible software, and they use various mechanisms to ensure that installed packages know how to find one another within an environment.
+Concretely, we can break down the purposes of package managers into five main tasks for the purpose of this discussion, namely ensuring that:
+1. packages are compatible with low-level system libraries and architecture, kernel drivers, etc.
+2. all packages know how to find each other
+3. all installed binaries know how to find their dependencies
+4. all packages have compatible application programming interfaces (APIs)
+5. all installed binaries have compatible application binary interfaces (ABIs)
+
+Some relevant notes on the above:
+- To address point 1, package managers must be able to query the system for the properties needed to satisfy the package's requirements.
+- To address points 2 and 3, package managers must install packages into predictable locations depending on the type of package. The distinction between these two points is that the loader is responsible for how binaries find their dependencies, while the language interpreters are responsible for ensuring that code written in those languages can find its dependencies, so the package manager must install packages in appropriate locations. Both the [GNU loader](https://man7.org/linux/man-pages/man8/ld.so.8.html) and [Python](https://docs.python.org/3/reference/import.html#searching) have well-documented procedures for finding Python modules.
+- To address points 4 and 5, package managers rely on package metadata to provide enough information to determine when two packages are compatible. At minimum, this comes in the form of package versions, but it can also include other information to, for example, identify the package's ABI.
+
+Although a general classification of package managers is challenging due to many of them serving multiple use cases, for this discussion we primarily care about how they solve the above problems.
+For this purpose, package managers typically come in two flavors:
+1. System package managers (e.g. `apt` or `yum` on Linux) manage a single centralized set of packages that are installed to standard system locations that are visible to all other packages and the loader. Only a single version of a package is typically available, so minimal dependency resolution is required, and the entire set of available packages is built from the ground up to be compatible, thereby satisfying all five criteria above.
+2. Third-party managers (Spack, conda, nix) typically have some way to produce isolated environments within which libraries installed by that package manager. These package managers support installing different versions of packages into different environments, so complex dependency resolution is required to resolve a given environment specification. Mechanisms like environment variables (such as `LD_LIBRARY_PATH` on Linux or `PYTHONPATH` in general) are used when environments are active to ensure that packages within the environment take precedence over system packages. Within a given environment, all libraries are handled consistently, satisfying all five criteria above.
+
+## Gaps in pip functionality
+
+With all of the above in mind, we can finally answer our primary question: why is it so difficult to distribute Python extension modules with pip?
+The fundamental challenge is that it does not fall into either category of package managers described above.
+At present, pip only satisfies criteria 2 and 4 above, and only for Python packages.
+It does not have the ability to query the system for the properties needed to satisfy 1 (e.g. it cannot account for C++ ABI, the CUDA version on the system, other drivers, etc), it does not install packages into system locations where the loader could find them to satisfy point 3, and the metadata that pip supports is insufficient to fully describe library ABIs since pip is primarily designed around Python packages where only API matters.
+Additionally, pip cannot be guaranteed to be used inside a virtual environment, so mechanisms involving e.g. setting environment variables are not guaranteed to work.
+Therefore, we must find ways to patch these gaps as best as we can using other means.

doc/source/reference/design.md

@@ -0,0 +1,21 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES.
+SPDX-License-Identifier: Apache-2.0
+
+Modified from original source to remove implementation-specific references.
+-->
+
+# Library Design
+
+Here we discuss design considerations for tools implementing these concepts.
+Tools implementing these concepts take the information from the [](#solutions) page but make opinionated choices about how to leverage them.
+This page aims to explain those choices, as well as provide context on how to safely deviate from those choices on a case-by-case basis.
+The discussion here assumes the library loading approach proposed in [](#solutions) (Option 3e).
+
+One architectural approach assumes that the package containing a library includes Python code to expose that library.
+The benefit of this approach is that the package can then provide a uniform interface regardless of how the files in the package are rearranged, and consumers can interact with this solely in Python.
+However, it is equally viable to put the onus on consumers to determine how the library should be loaded.
+In that case, the library wheel can be packaged essentially as-is.
+Consuming wheels would be responsible for performing the `ctypes` calls themselves.
+The primary downside of this approach is that the same logic has to be duplicated in many places.
+The benefit is that the library wheel can be packaged essentially as-is, particularly if the approach taken is a simple binary repackaging.

doc/source/reference/index.rst

@@ -0,0 +1,19 @@
+Mechanics of Shared Library Distribution
+========================================
+
+The way that shared libraries work is a deep and complex topic, so this document aims to provide sufficient high-level background and copious links to useful resources for the interested reader to go deeper.
+The rest of the document concretely outlines the various mechanisms we could leverage for sharing libraries and the pros and cons of each approach.
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   background
+   solutions
+   design
+
+.. note::
+   The technical reference documentation in this section (Background, Solutions, and Design pages) is derived from the `native_lib_loader project <https://github.com/wheelnext/native_lib_loader>`__, originally authored by Vyas Ramasubramani.
+
+   Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES.
+   Licensed under the Apache License, Version 2.0.