Skip to content

Modern Tooling + Parallel Vault loading#55

Closed
JavierGOrdonnez wants to merge 9 commits intomfarragher:mainfrom
JavierGOrdonnez:introduce-parallel-loading
Closed

Modern Tooling + Parallel Vault loading#55
JavierGOrdonnez wants to merge 9 commits intomfarragher:mainfrom
JavierGOrdonnez:introduce-parallel-loading

Conversation

@JavierGOrdonnez
Copy link

This is a follow-up PR from #54 (so it includes those changes - modern python tooling: uv, ruff, ...) and additionally introduces parallelism in the Vault generation:

  • ProcessPool ensures true parallelism - ThreadPool failed to speed up due to GIL
  • Backwards-compatible: not providing the "workers" parameter at Vault initialization defaults to previous sequential behaviour
  • Fully tested against sequential loading

On a 12-core AMD processor:

  • medium-sized vault (500 markdown files): 8 workers provided 2x speedup on connect() and 1.5x on gather()
  • large-sized vault (2.000 markdown files): 8 workers provided 4-5x speedup, 12/16 workers provided 5-6x speedup on both connect() and gather()

claude and others added 9 commits March 5, 2026 13:16
- Replace setup.py + requirements*.txt with a single pyproject.toml
  using hatchling as build backend
- Add [tool.uv] dev-dependencies for uv-managed dev workflow
- Add [tool.ruff] config for linting (E/W/F/I/UP/B/C4/SIM) and
  formatting (replaces any prior flake8/black/isort setup)
- Add [tool.ty] config for static type checking with ty
- Add [tool.pytest.ini_options] and [tool.coverage] sections
- Update CI to use astral-sh/setup-uv and uv sync/run, upgrade
  actions/checkout to v4, add dedicated lint and typecheck jobs

uv.lock will be generated automatically on first `uv sync`.

https://claude.ai/code/session_01G13wzGLvmoog4E73fUairt
- Fix F401: use explicit re-export syntax in __init__.py (as api, etc.)
- Fix I001: sort all import blocks to isort standard
- Fix C416/C404/C405: replace unnecessary comprehensions with simpler
  equivalents (dict(), list(), set literals)
- Fix SIM118: remove .keys() in dict membership tests
- Fix UP032: convert .format() calls to f-strings
- Fix B006: replace mutable default arg with None sentinel in
  html_processing._remove_main_formatting
- Fix B028: add stacklevel=2 to warnings.warn in api.py
- Fix E722: replace bare except with except Exception in md_utils.py
- Fix W291: suppress intentional trailing whitespace in test string literal
  via per-file-ignores in pyproject.toml
- Run ruff format over all 23 files (consistent style)
- Remove --cov flags from pytest addopts (added by CI, not hardcoded)

https://claude.ai/code/session_01G13wzGLvmoog4E73fUairt
…n-malformed-file

Add skipping avoid crash on malformed file
@JavierGOrdonnez JavierGOrdonnez changed the title Moder Tooling + Parallel Vault loading Modern Tooling + Parallel Vault loading Mar 12, 2026
@JavierGOrdonnez JavierGOrdonnez deleted the introduce-parallel-loading branch March 13, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants