Missing YAML config files in installed package

[jalengg]

Problem

When PyHealth is installed via pip (from git or PyPI), the YAML configuration files in pyhealth/datasets/configs/ are not included in the installed package. This causes FileNotFoundError when trying to use several datasets.

Error Example

from pyhealth.datasets import HALO_MIMIC3Dataset

dataset = HALO_MIMIC3Dataset(mimic3_dir="data")

Results in:

FileNotFoundError: [Errno 2] No such file or directory: 
'/usr/local/lib/python3.12/dist-packages/pyhealth/datasets/configs/hcup_ccs_2015_definitions_benchmark.yaml'

Affected Datasets

This issue affects multiple datasets that rely on YAML configuration files:

• halo_mimic3.py → hcup_ccs_2015_definitions_benchmark.yaml
• mimic3.py → mimic3.yaml
• mimic4.py → mimic4_cxr.yaml, mimic4_ehr.yaml, mimic4_note.yaml
• ehrshot.py → ehrshot.yaml
• covid19_cxr.py → covid19_cxr.yaml
• medical_transcriptions.py → medical_transcriptions.yaml

Root Cause

The setup.py has include_package_data=True but there is no MANIFEST.in file to specify which non-Python files should be included in the package. By default, setuptools only includes .py files.

Solution

Create a MANIFEST.in file in the repository root:

include README.rst
include requirements.txt
include LICENSE
recursive-include pyhealth/datasets/configs *.yaml *.yml

This tells setuptools to include all YAML files in the configs directory when building the package.

Verification

After the fix, verify with:

python setup.py sdist
tar -tzf dist/pyhealth-*.tar.gz | grep "\.yaml"

Should show:

pyhealth-1.1.4/pyhealth/datasets/configs/covid19_cxr.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/ehrshot.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/hcup_ccs_2015_definitions_benchmark.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/medical_transcriptions.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/mimic3.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/mimic4_cxr.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/mimic4_ehr.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/mimic4_note.yaml

Context

Discovered while fixing the HALO Colab notebook in PR sunlabuiuc#528. The notebook installs PyHealth from git and users encountered this error when trying to load the HALO_MIMIC3Dataset.

This is a project-wide packaging issue that affects any user installing PyHealth via pip rather than running from source.

[jalengg]

Detailed Analysis and Rationale

Investigation

I traced through the error reported in the Colab notebook where HALO_MIMIC3Dataset failed with:

FileNotFoundError: [Errno 2] No such file or directory: 
'/usr/local/lib/python3.12/dist-packages/pyhealth/datasets/configs/hcup_ccs_2015_definitions_benchmark.yaml'

The file exists in the repository at pyhealth/datasets/configs/hcup_ccs_2015_definitions_benchmark.yaml, but when PyHealth is installed via pip install git+https://github.com/..., the YAML files are missing from the installed package.

Root Cause

The setup.py file already has include_package_data=True (line 47), which is designed to include non-Python files in the package. However, this flag requires a MANIFEST.in file to specify which non-Python files to include. Without MANIFEST.in, setuptools defaults to only including .py files.

Scope of Impact

This is not just a HALO issue. I verified that 6 different dataset modules rely on these YAML config files:

1. halo_mimic3.py (line 78) → hcup_ccs_2015_definitions_benchmark.yaml
2. mimic3.py (line 16) → mimic3.yaml
3. mimic4.py (multiple lines) → mimic4_cxr.yaml, mimic4_ehr.yaml, mimic4_note.yaml
4. ehrshot.py (line 6) → ehrshot.yaml
5. covid19_cxr.py (line 2) → covid19_cxr.yaml
6. medical_transcriptions.py (line 12) → medical_transcriptions.yaml

All of these datasets would fail with FileNotFoundError when installed via pip.

Why MANIFEST.in is the Right Solution

I considered three approaches:

Option 1: MANIFEST.in (chosen)

• Standard approach for projects using setup.py with include_package_data=True
• Separates concerns: setup.py defines code structure, MANIFEST.in defines data files
• Most maintainable and conventional
• No changes needed to existing setup.py

Option 2: package_data in setup.py

• Would require modifying setup.py to add package_data={'pyhealth.datasets': ['configs/*.yaml']}
• Less conventional when include_package_data=True is already set
• Mixes code structure with data file declarations

Option 3: Migrate to pyproject.toml

• More modern but requires significant refactoring
• Overkill for fixing this immediate issue
• Could be done later as separate work

Verification

I built a source distribution and confirmed all 8 YAML files are now included:

$ python setup.py sdist
$ tar -tzf dist/pyhealth-1.1.4.tar.gz | grep "datasets/configs/.*\.yaml$"
pyhealth-1.1.4/pyhealth/datasets/configs/covid19_cxr.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/ehrshot.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/hcup_ccs_2015_definitions_benchmark.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/medical_transcriptions.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/mimic3.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/mimic4_cxr.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/mimic4_ehr.yaml
pyhealth-1.1.4/pyhealth/datasets/configs/mimic4_note.yaml

Next Steps

Currently investigating whether this fix resolves the Colab notebook error. Will test by:

1. Installing from the updated branch in a fresh Colab environment
2. Running the HALO_MIMIC3Dataset initialization
3. Confirming the YAML file is accessible in the installed package

Status: Investigating/Testing

MANIFEST.in Contents

include README.rst
include requirements.txt
include LICENSE
recursive-include pyhealth/datasets/configs *.yaml *.yml

This is minimal and only includes files that are actually referenced by the codebase.

[jalengg]

Update: Previous Fix Was Wrong — New Fix Applied (commit 6ce060d)

Why MANIFEST.in Didn't Work

The previous fix (commit 2d4fbbd, MANIFEST.in) was incorrect for this use case. Here's why:

• MANIFEST.in controls what goes into sdist (source distribution) tarballs — i.e., python setup.py sdist or pip download.
• When Colab runs pip install git+https://..., pip clones the repo and runs pip install . directly — it does not use MANIFEST.in at all.
• For direct installs, what matters is package_data in setup.py.

Root Cause (Confirmed)

setup.py had include_package_data=True but no package_data entry. The include_package_data=True flag is supposed to include VCS-tracked non-Python files, but in practice it's unreliable without an explicit package_data declaration — especially under pip's build isolation environment.

The YAML files are tracked by git (git ls-files confirms them), but include_package_data=True alone was not sufficient.

Actual Fix

Added explicit package_data to setup.py:

package_data={"pyhealth": ["datasets/configs/*.yaml"]},

Removed MANIFEST.in as it provided no benefit for pip install git+https://... installs.

Commit

6ce060d1 — "Fix YAML config packaging: use package_data instead of MANIFEST.in"

Status: Investigating/Testing in Colab

[jalengg]

Update: Commits Pushed to halo-pr-528

All fixes are now live on jalengg/PyHealth @ halo-pr-528:

Commit	Description
6ce060d1	Fix YAML config packaging: use package_data instead of MANIFEST.in
663dbb8d	Add last-updated timestamp to Colab notebook header

The key fix (6ce060d1) adds explicit package_data to setup.py:

package_data={"pyhealth": ["datasets/configs/*.yaml"]},

This is the correct mechanism for pip install git+https://... installs (e.g. Colab). MANIFEST.in was also removed as it only affects sdist builds and provided no benefit here.

Status: Ready for testing in Colab — install from halo-pr-528 and confirm HALO_MIMIC3Dataset loads without FileNotFoundError.