v0.8.4

v0.8.4

Added

• GOALS.md documenting project goals: local ancestry ownership, tree visualization, sparse trees, and cross-platform data curation

v0.8.3

v0.8.3

Added

• 12 shared server utilities (paths, downloadImage, parseYear, lifespan, validation, asyncHandler, sseHelpers, operationTracker, resolveCanonical, browserConnect, applyOverrides, batchFetchPersons)
• useBrowserConnection and useSSE client hooks for deduplication
• Path traversal defense-in-depth across all file-serving endpoints
• FTS5 query sanitization for search
• CORS origin restriction via CORS_ORIGIN env var
• Request body whitelisting on provider/browser routes
• URL validation replacing string includes() checks
• --dry-run flag for purge script
• AbortController cleanup on PersonDetail navigation
• PWA support: favicon, manifest, apple-touch-icon, theme-color

Changed

• CI no longer auto-increments version on push to dev
• Release workflow no longer bumps version after release (changelog archiving preserved)
• CI permissions downgraded from contents: write to contents: read
• process.exit(1) in fetcher replaced with typed error
• ID mapping cache changed from FIFO to LRU eviction
• AI discovery runs bounded with MAX_STORED_RUNS limit
• Path finding guarded against infinite loops
• Error handler now logs stack traces
• requestTimeout middleware wired up
• Person.parents type corrected to (string | null)[]
• Scripts standardized on import.meta.dirname and shared types

Fixed

• person_computed view multi-row bug in vital event joins
• typeof gen !== '?' bug in logPerson
• Stale closures in IntegrityPage loader functions
• D3 zoom cleanup missing in SparseTreePage
• useSocket spread in deps array causing re-renders
• useSocketConnection returning stale snapshot instead of reactive state
• Dead state (setScrapedData, setImportFile) removed
• Safe JSON.parse on localStorage in PersonDetail
• Purge script whole-token matching (prevents partial ID matches)
• Migration script wrapped in transaction

Removed

• 20+ duplicate DATA_DIR declarations across services
• 3 duplicate downloadImage implementations
• 7 duplicate SSE header setup blocks
• 15 duplicate ULID validation blocks in person routes
• 3 duplicate browser connection blocks
• 3 duplicate operation tracker implementations
• 6 duplicate photo-serve route handlers
• Duplicate BrowserStatus interface in IndexerPage

v0.7.12

Release v0.7.12 - Migration Map, AI Discovery Improvements

Released: Unreleased

Overview

This release improves the AI Discovery feature with dismiss functionality, debugging capabilities, and better CLI integration.

New Features

AI Discovery Dismiss Functionality (v0.7.0)

• Dismiss AI-discovered candidates to exclude them from future discoveries
• Batch dismiss all unselected candidates at once
• View dismissed candidates with original AI reasoning and tags
• Undo dismiss to restore candidates
• Clear all dismissed candidates for a database
• Dismissed candidates stored in SQLite with discovery_dismissed table
• API endpoints: POST /dismiss, POST /dismiss-batch, GET /dismissed, DELETE /dismissed/:personId, DELETE /dismissed

AI Discovery Debug Endpoints (v0.7.0)

• New debug endpoints for troubleshooting AI run failures
• GET /api/ai-discovery/debug/runs - list recent AI runs with metadata
• GET /api/ai-discovery/debug/runs/:runId - detailed run info including prompt preview and output
• Error details capture (last 10 lines of stderr/stdout) stored on failed runs

Migration Map Visualization (v0.7.12)

• New 6th tree view mode: "Migration Map" plots ancestors on an interactive world map
• Leaflet.js map with OpenStreetMap/CartoDB dark tiles
• Lineage-colored markers (paternal=blue, maternal=red) with generation depth variation
• Migration polylines connecting parent birth to child birth locations
• Time range slider for filtering by birth year
• Paternal/Maternal layer toggle checkboxes
• Geocoding via Nominatim with permanent SQLite cache (place_geocode table)
• Progressive geocode broadening: strips specific locality parts to resolve historical place names
• User-initiated batch geocoding with SSE progress streaming (EventSource)
• Sparse tree map page at /favorites/sparse-tree/:dbId/map
• API: GET /api/map/:dbId/:personId, GET /api/map/:dbId/sparse, GET /api/map/geocode/stream

Bug Fixes

CLI Provider Integration (v0.7.0)

• Fixed large prompt handling via stdin piping for CLI providers (codex, claude-code)
• Added - flag detection in provider args to enable stdin mode
• Fixed JSON parsing to find LAST valid JSON array in noisy CLI output
• Handles CLI banners and headers that could interfere with response parsing
• Improved error capture with errorDetails field for debugging failures

v0.6.33

Release v0.6.33 - FamilySearch Normalization & Architecture Refinement

Released: YYYY-MM-DD

Overview

This release normalizes FamilySearch as a downstream provider (equal to Ancestry, WikiTree, etc.) and establishes SparseTree SQLite as the canonical source of truth. Major architectural cleanup removes all legacy fallback code in favor of migration scripts.

New Features

Ancestry Update Automation Page (v0.6.33)

• New database-scoped page at /db/:dbId/ancestry-update for bulk Ancestry synchronization
• BFS traversal of direct-ancestor sparse tree from selected root person
• Configurable generation depth (1-4 or full tree)
• Per-person pipeline: check Ancestry link, process free hints, download/scrape provider data
• Real-time SSE progress with emoji-prefixed execution log
• Test mode (dry run) for previewing what would happen
• Adds "Ancestry Update" link to database sidebar
• API endpoints: GET /api/ancestry-update/status, GET /:dbId/events (SSE), POST /:dbId/cancel, GET /:dbId/validate/:personId

Ancestry Free Hints Automation (v0.6.33)

• New "Hints" button on Ancestry row in Provider Data table to process free record hints
• Playwright automation navigates to hints page, reviews each hint, accepts and saves to tree
• Supports checkbox selection for adding related people from hints
• Includes consecutive failure safeguard (stops after 3 failures) and remaining hints re-check
• SSE-based progress events for real-time UI updates
• API endpoints: POST/GET /api/ancestry-hints/:dbId/:personId, cancel, status

Ancestry Vital Info Upload (v0.6.33)

• Extended Ancestry upload to support birth date, birth place, death date, death place, and name (not just photos)
• Upload dialog now shows field-by-field comparison between local SparseTree data and cached Ancestry data
• Browser automation uses Ancestry's Quick Edit sidebar to update person facts
• Automatically handles Living/Deceased status toggle when setting death information
• Pre-selects all differing fields for upload convenience

FamilySearch Normalization (v0.6.5)

• FamilySearch now treated as equal downstream provider (no special priority)
• Photos use -familysearch suffix like other providers: {personId}-familysearch.jpg
• Added linkFamilySearch() and getFamilySearchPhotoPath() to augmentation service
• ID resolution uses alphabetical order (no FamilySearch priority)

Architecture: Source of Truth Principle (v0.6.33)

• SparseTree SQLite is now the canonical source of truth
• Provider cache (data/provider-cache/) is for comparison only, not a fallback data source
• All data reads come from SQLite; provider cache shows what providers have vs what SparseTree has

Bug Fixes

SQLite Vital Event Row ID Preservation (v0.6.33)

• Changed vital_event INSERT from INSERT OR REPLACE to ON CONFLICT DO UPDATE
• Preserves row IDs which are critical for local_override references
• INSERT OR REPLACE was deleting and re-inserting rows, orphaning override references

Improved Vital Event Date Parsing (v0.6.33)

• Now uses actual birth.date and death.date objects instead of parsing lifespan string
• Captures full dates (e.g., "1979-07-31") rather than just years from lifespan ("1979-2020")
• Added parseYearFromDate() function to handle various date formats including BC notation

Ancestry Update Log Memory Fix (v0.6.33)

• Limited execution log entries to 500 to prevent browser memory issues on large trees
• Shows notice when older entries are truncated

FamilySearch Upload Improvements (v0.6.33)

• Auto-refresh FamilySearch cache after successful upload so UI shows updated data
• Improved photo comparison to detect already-uploaded photos via symlink matching
• Fixed Google login button selector for updated FamilySearch login page
• Improved vitals page editing: handles conclusion dialogs and edit buttons
• Added autocomplete handling for date/place fields (clicks first suggestion)

Scraper Auto-Login (v0.6.33)

• Scraper now attempts auto-login via Google OAuth when landing on FamilySearch login page
• Added handleLoginIfNeeded() function to detect login pages and complete OAuth flow
• Navigates directly to target URL after successful login

Provider Comparison Not Using Local Overrides (v0.6.33)

• Fixed comparison service using original database values instead of user-applied overrides
• Added applyLocalOverrides() to fetch and apply vital_event overrides before comparison
• FamilySearch/Ancestry now correctly show "match" when user has applied their values

Date Format Differences Showing as Different (v0.6.33)

• Added date normalization to expand month abbreviations (e.g., "AUG" → "august")
• "29 AUG 1933" now correctly matches "29 August 1933"
• Reduces false positives in provider comparison diff counts

Unicode Escape in View Mode Selector (v0.6.33)

• Fixed checkmark showing as literal \u2713 text instead of ✓ in tree view mode dropdown
• JSX text content requires actual Unicode characters, not escape sequences

Fan Chart Ancestor Count (v0.6.33)

• Fixed footer showing incorrect count like "109 of 62 ancestors shown"
• Now correctly counts only ancestors within the visible generation range

Stale Browser Connection Detection (v0.6.33)

• Added verifyAndReconnect() to browser service that checks both Playwright state and actual browser process
• Augmentation service now detects and recovers from stale connections before scraping
• Fixes silent failures when browser was closed but connection state was still "connected"

Auto-Login on Sync (v0.6.33)

• FamilySearch sync now attempts automatic login when auth token is missing
• Uses providerService.ensureAuthenticated() to try Google SSO or stored credentials
• Eliminates need for manual browser login before sync operations
• Applies to both refreshPerson() and fetchPersonDisplayName() methods

FamilySearch Scraper Not Extracting Data (v0.6.33)

• Updated scraper selectors to match current FamilySearch DOM structure
• Name now extracted using [data-testid="fullName"]
• Birth/death dates now extracted from [data-testid="conclusionDisplay:BIRTH"] and [data-testid="conclusionDisplay:DEATH"] containers
• Added handling for collapsed Vitals section (auto-expands before extracting)
• Photo extraction uses avatar container near [data-testid="update-portrait-button"]
• Fixed TypeScript __name decorator issue causing page.evaluate to fail in browser context
• Added detailed debug logging for photo extraction troubleshooting
• Improved UI feedback: loading toast during scrape, success toast shows extracted fields

"Use" Button Not Updating SparseTree Row (v0.6.33)

• Fixed clicking "Use" on provider field values not updating the SparseTree row display
• Root causes fixed:
  • Client: onFieldChanged callback now refetches person data after applying override
  • Server: getPerson() now applies local overrides to returned person data
• Added applyLocalOverridesToPerson() to database service for consistent override application
• Birth/death date/place values from providers now properly display after clicking "Use"

"Set as Primary" Photo Not Updating (v0.6.33)

• Fixed photo not updating after clicking "Set as Primary" on provider photos
• Root causes fixed:
  • getPhotoPath() now checks for primary photo ({personId}.jpg) before provider-suffixed photos
  • Photo priority in PersonDetail header and SparseTree row now shows primary photo first
  • Added Cache-Control: no-cache headers to all photo "exists" endpoints to prevent stale responses
• Added FamilySearch-specific photo endpoint: GET /augment/:personId/familysearch-photo
• Added client API methods: getFsPhotoUrl(), hasFsPhotoProvider()
• Added photoVersion state for cache-busting photo URLs after changes

HMR Context Resilience (v0.6.33)

• Fixed "useSidebar must be used within a SidebarProvider" error during HMR transitions
• useSidebar() and useTheme() hooks now return safe defaults during dev hot reloads
• Prevents crashes when React context identity changes during development

Improvements

FamilySearch Upload Performance (v0.6.33)

• Eliminated redundant post-login delays in FamilySearch upload automation
• After Google OAuth redirect, now checks if already on target page before re-navigating
• Replaced fixed waitForTimeout delays with element-based waitForSelector for responsiveness
• Added PM2 timestamps to server logs for better debugging (time: true in ecosystem config)

Page-Level Padding Control (v0.6.33)

• Moved p-6 padding from Layout to individual pages for flexibility
• Full-height pages (tree views, PersonDetail, SparseTreePage) control their own layout
• Standard content pages receive consistent padding via their root element
• Fixed missing padding on SparseTreePage and PersonDetail
• Removed double padding from AIProviders page (external component handles its own padding)

Tree Visualization Parity with Ancestry.com (v0.6.33)

• Added 5 tree view modes matching Ancestry.com functionality
• Fan Chart (default): Radial pedigree with lineage-colored wedges (paternal=blue/teal, maternal=red/coral)
• Horizontal Pedigree: Root left, ancestors right with expandable nodes
• Vertical Family: Ancestors top, root middle with generation labels
• Columns and Focus: Existing views retained
• URL-based routing: /tree/:dbId/:personId/:viewMode
• New shared components: TreeCanvas (D3 zoom/pan), TreeControls, AncestorNode
• New utilities: lineageColors, treeLayout, arcGenerator
• Fixed classic view bugs: dynamic centering, ResizeObserver instead of setTimeout

Vertical Family View Improvements (v0.6.33)

• Refactored layout to use multi-pass collision resolution algorithm
• Parents properly avoid overlapping when multiple branches are expanded
• Added horizontal coupling bars between parent pairs (matching Ancestry.com style)
• Fixed expand button centering on cards
• Ex...

v0.5.4

Release v0.5.4 - Multi-Platform Comparison & UI Redesign

Released: 2026-01-25

Overview

This release adds multi-platform data comparison capabilities, allowing you to compare person data across FamilySearch, Ancestry, WikiTree, and Wikipedia. It also includes a major UI redesign for the person detail page with a more compact, user-friendly layout.

New Features

Multi-Platform Data Comparison (v0.5.0)

• Compare person data across multiple genealogy providers side-by-side
• Field-by-field comparison with visual indicators: ✓ match, ⚠ different, — missing
• Unified provider cache structure at data/provider-cache/{provider}/{externalId}.json
• Download/refresh data from providers directly from the UI
• Collapsible field comparison panel showing divergences

Compact PersonDetail Layout (v0.5.0)

• Redesigned person detail page with more efficient use of screen space
• Side-by-side Vital Events (birth/death/burial) and Family cards at top
• Inline tags for Also Known As and Occupations
• Unified Platforms & Data section combining provider links and comparison
• Compact bio section

Link Platform Dialog (v0.5.0)

• New modal dialog for linking Wikipedia, Ancestry, and WikiTree profiles
• Platform-specific styling and placeholder text
• Keyboard support (Escape to close, Enter to submit)
• Auto-focus on URL input when opened

Ancestry Scraper Updates (v0.5.0)

• Updated selectors for 2024/2025 Ancestry DOM structure
• Extracts name from .userCardTitle
• Extracts birth/death from .userCardEventDetail spans
• Extracts parents from Relationships section
• Extracts profile photo URL
• Proper handling of "Unknown" and "Living" status values

New Files

Client Components

• client/src/components/person/ProviderDataTable.tsx - Unified provider data table with photos, status, and actions
• client/src/components/person/UnifiedPlatformSection.tsx - Combined provider links and comparison table (replaced by ProviderDataTable)
• client/src/components/person/LinkPlatformDialog.tsx - Modal dialog for linking platforms
• client/src/components/person/VitalEventCard.tsx - Compact vital event display with editing
• client/src/components/ui/EditableDate.tsx - Inline date editor with genealogy format validation
• client/src/components/ui/EditableField.tsx - Inline text field editor
• client/src/components/ui/EditableList.tsx - Inline list editor

Server Services

• server/src/services/multi-platform-comparison.service.ts - Provider data comparison logic
• server/src/services/familysearch-refresh.service.ts - FamilySearch data refresh
• server/src/services/familysearch-upload.service.ts - Upload changes to FamilySearch
• server/src/services/local-override.service.ts - Local override management

Scripts

• scripts/migrate-provider-cache.ts - Migration script for unified cache structure

API Endpoints

New Endpoints

• GET /api/sync/:dbId/:personId/multi-platform-compare - Get full multi-platform comparison
• POST /api/sync/:dbId/:personId/refresh-provider/:provider - Refresh data from specific provider

Technical Details

Provider Cache Structure

Provider data is now cached in a unified structure:

data/provider-cache/
  familysearch/{fsId}.json
  ancestry/{externalId}.json
  wikitree/{externalId}.json

Comparison Algorithm

• FamilySearch serves as the baseline (primary source)
• Field comparison uses fuzzy matching for places (contains check)
• Alternate names are sorted alphabetically before comparison to avoid false differences
• Missing values are distinguished from empty values

Provider Data Table (v0.5.2)

• New unified ProviderDataTable component replacing UnifiedPlatformSection
• SparseTree shown as primary row with all provider data below
• Columns: Source, Photo, Name, Birth, Death, Father, Mother, Children, AKA, Occupations, Status, Actions
• Photo thumbnails with hover "Use Photo" overlay
• Download buttons now include photo fetching automatically
• Upload to FamilySearch dialog with photo upload option via Playwright automation

Compact Header & Vitals (v0.5.2)

• Redesigned header: smaller photo (24x24), inline edit button next to name
• IDs shown inline without expand/collapse toggle
• Vital events condensed to single row (Birth • Death • Burial)
• Parents and children shown in compact row below vitals
• Also Known As names now alphabetically sorted

Parent Provider ID Discovery (v0.5.3)

• Automatically discover and link provider-specific IDs for a person's parents
• Scraper extractParentIds method added to FamilySearch, Ancestry, and WikiTree scrapers
• Parent discovery service with name matching (accent normalization, partial match)
• Single-person discovery and BFS ancestor traversal modes
• API endpoints: POST /api/sync/:dbId/:personId/discover-parents/:provider and discover-ancestors/:provider
• Discover buttons in ProviderDataTable with per-row and "Discover All" BFS options
• parentsNeedDiscovery field on provider links for smart UI hints

Ancestry Photo Upload (v0.5.3)

• Upload local photos to Ancestry via Playwright browser automation
• ancestry-upload.service.ts: compare, upload, auto-login with stored credentials
• UploadToAncestryDialog.tsx: emerald-themed upload dialog with photo preview
• API endpoints: compare-for-ancestry-upload and upload-to-ancestry
• Upload button on Ancestry row in ProviderDataTable

Login Polling Fix (v0.5.3)

• Replaced fixed 5-second waitForTimeout after login submit with 500ms polling loop
• Google OAuth and fast logins now detected within ~500ms instead of waiting full 5 seconds
• Polls checkLoginStatus and error indicators every 500ms, up to 15s max timeout
• Applies to all providers using performLoginWithSelectors (FamilySearch, Ancestry, WikiTree, 23andMe)

Dead Code Removal (v0.5.3)

• Deleted UnifiedPlatformSection.tsx (superseded by ProviderDataTable)

Data Integrity Page + Bulk Discovery (v0.5.4)

• Database maintenance dashboard at /db/:dbId/integrity with summary cards and tabbed interface
• Integrity service: provider coverage gaps, parent linkage gaps, orphaned edges, stale cache detection
• Bulk discovery service: database-wide automated parent ID discovery with SSE progress and cancellation
• Parents tab: provider selector, "Discover All" button with real-time progress bar
• Coverage tab: persons with incomplete provider links shown with linked/missing badges
• Orphans tab: broken parent edges with missing person indicators
• Stale tab: configurable age threshold with color-coded staleness
• API endpoints at /api/integrity/:dbId for all checks and bulk operations
• Sidebar nav item with ShieldCheck icon under each database

Full Changelog

Full Diff: v0.4.16...v0.5.0

v0.4.16

Release v0.4.16 - AI Discovery & SQLite Stability

Released: 2026-01-23

Overview

This release adds AI-powered ancestor discovery, test coverage reporting UI, and critical SQLite stability fixes to prevent database lock contention during concurrent operations.

New Features

AI Discovery & Test Coverage (v0.4.0)

• Added AI discovery feature for finding notable ancestors
• Test coverage reports UI for viewing test status

SQLite Concurrency Fixes (v0.4.2)

• Fixed database lock contention causing app freezes during concurrent indexing and queries
• Added busy_timeout = 30000 pragma - waits up to 30s for locks instead of failing immediately
• Added wal_autocheckpoint = 1000 pragma - auto-checkpoints at ~4MB to prevent WAL bloat
• Added checkpoint() function to sqliteService for manual WAL checkpointing
• Added automatic WAL checkpoint after finalizeDatabase() in sqlite-writer

Bug Fixes

Fixed: App Freezing During Indexing (v0.4.2)

• Symptom: App became unresponsive when running path calculations while indexer was active
• Cause: WAL file grew to 1.2GB causing SQLite lock contention between reader and writer processes
• Fix: Added proper busy_timeout and wal_autocheckpoint pragmas, plus periodic checkpointing

Fixed: Sparse Tree Page Freezing (v0.4.2)

• Symptom: App froze when navigating to favorites sparse-tree page on large databases
• Cause: Service was loading entire database (138k+ persons) into memory for BFS path finding
• Fix: Rewrote sparse-tree service to use SQLite queries for path finding with iterative BFS

Fixed: Relationship Calculation Freezing (v0.4.2)

• Symptom: App froze when calculating relationship between two people
• Cause: Recursive CTEs explored exponentially many paths; fallback loaded entire database
• Fix: Rewrote path service to use iterative BFS with common ancestor algorithm (0.3s vs freeze)

Technical Details

The SQLite WAL (Write-Ahead Log) mode provides excellent read concurrency, but can cause issues when:

1. The WAL file grows very large (>100MB)
2. Multiple processes access the database simultaneously
3. No busy_timeout is set (causing immediate failures instead of retries)

The fixes ensure:

• Readers wait for writers instead of failing immediately
• WAL is regularly checkpointed to prevent unbounded growth
• Large batch operations trigger explicit checkpoints

Full Changelog

Full Diff: v0.3.21...v0.4.2

v0.3.24

Release v0.3.24

Changes

• fix: separate playwright configs for scraper vs e2e tests
• fix: correct cache path for shared package (types/ not dist/)
• fix: use tsx for server start to fix ES module resolution
• ci: simplify triggers to avoid duplicate runs
• refactor: consolidate CI and test workflows into single pipeline
• fix: store commit SHA before cherry-pick in release workflow
• fix: add build step to scraper tests in CI workflow

Installation

git clone https://github.com/atomantic/SparseTree.git
cd SparseTree
npm run install:all
pm2 start ecosystem.config.cjs

v0.3.21

Release v0.3.21 - SQLite Storage Layer & Migration Framework

Released: YYYY-MM-DD

Overview

Major architectural upgrade introducing SQLite as a high-performance index layer while maintaining JSON files as the source of truth. This release adds canonical ULID-based identities, full-text search via FTS5, recursive CTEs for path finding, and a comprehensive data migration framework.

🎉 New Features

SQLite Storage Layer

• Added SQLite database (data/sparsetree.db) as fast query index
• FTS5 full-text search for person names, bios, and occupations
• Recursive CTEs for efficient ancestor/descendant path finding
• WAL mode for better read concurrency
• Auto-enables when database exists with data, falls back to JSON otherwise

Canonical Identity System

• ULID-based canonical IDs (26-char, sortable, collision-resistant)
• External identity mappings for multiple providers (FamilySearch, Ancestry, WikiTree, 23andMe)
• Bidirectional ID lookup with in-memory LRU cache
• Confidence scoring for identity assertions

Content-Addressed Blob Storage

• SHA-256 hash-based media storage (data/blobs/{hash[:2]}/{hash}.{ext})
• Automatic deduplication of duplicate photos
• Media records linked to persons with primary photo support

Data Migration Framework

• Automatic schema and data migrations on update
• Migration tracking in data/.data-version and SQLite migration table
• Dry-run mode for previewing changes
• Rollback support for reversible migrations
• Commands: npm run migrate, npm run migrate:status, npm run migrate:dry-run

Update Script

• New ./update.sh for one-command updates
• Pulls latest code, installs deps, builds, migrates, restarts
• Supports --dry-run, --no-restart, --branch=NAME options
• Safe: checks for uncommitted changes before updating

Simplified Genealogy Provider UI (v0.3.10)

• Consolidated genealogy provider management to a single /providers/genealogy page
• Removed over-engineered /providers/scraper and /providers/genealogy/:id/edit routes
• Simplified BrowserSettingsPage to focus on CDP connection settings only
• Added "Login with Google" SSO option for FamilySearch
• Browser connection controls now available directly on the providers page
• Provider credentials and auto-login moved to the consolidated page

Fixed Database Refresh Hang (v0.3.11)

• Fixed refresh button hanging the server on large trees (138k+ persons)
• Changed recursive CTE query to use database_membership table when available
• Added hard limit (50 generations, 500k persons) to prevent runaway queries
• Converted refresh to SSE-based background task to avoid HTTP timeouts
• Client now uses EventSource for non-blocking refresh with progress updates

Documentation Restructure (v0.3.12)

• Created docs/ folder with modular documentation:
  • architecture.md - Data model, storage, identity system
  • api.md - API endpoint reference
  • cli.md - CLI command reference
  • development.md - Development setup guide
  • providers.md - Genealogy provider configuration
  • roadmap.md - Detailed phase documentation
• Simplified PLAN.md to high-level summary with links
• Streamlined CLAUDE.md with quick reference format
• Simplified README.md with user-focused content
• Added Phase 17 (Socket.IO real-time event system) to roadmap

Project Structure Consolidation (v0.3.13)

• Moved root lib/ directory into server/src/lib/:
  • lib/config.ts - Application configuration with TypeScript types
  • lib/graph/ - Path finding algorithms (shortest, longest, random)
  • lib/familysearch/ - FamilySearch API client, fetcher, and transformer
  • lib/sqlite-writer.ts - SQLite dual-write logic during indexing
• Moved root CLI scripts to scripts/:
  • index.ts - Main ancestry indexer
  • find.ts - Path finder
  • print.ts - Chronological print
  • purge.ts - Purge cached records
  • prune.ts - Prune orphan files
  • rebuild.ts - Rebuild databases
  • migrate-favorites.ts - Favorites migration
• Added server/src/utils/sleep.ts and randInt.ts utilities
• Updated indexer.service.ts to spawn npx tsx scripts/index.ts
• Removed deprecated tsv.js (redundant with /api/export/:dbId/tsv)
• All scripts now run via npx tsx for TypeScript support

Socket.IO Real-Time Events (v0.3.14)

• Implemented Socket.IO for bidirectional real-time communication
• Added server/src/services/socket.service.ts with room-based event broadcasting
• Added client/src/services/socket.ts singleton socket client
• Added client/src/hooks/useSocket.ts with React hooks for socket events:
  • useSocketConnection() - Manage socket connection lifecycle
  • useSocketEvent() - Subscribe to specific socket events
  • useDatabaseEvents() - Database refresh notifications
  • useBrowserEvents() - Browser status updates
  • useIndexerEvents() - Indexing progress
• Converted database refresh from SSE to Socket.IO events
• Browser status broadcasts via Socket.IO in addition to SSE (backwards compatible)
• Added in-memory LRU cache for SQL queries (server/src/services/cache.service.ts):
  • Configurable TTL and max size
  • Separate caches for queries, persons, and lists
  • Cache invalidation per database or person
  • Cache statistics for monitoring
• Optimized /api/favorites endpoint:
  • Replaced N+1 queries with single JOIN query
  • Removed loading of entire database contents
  • Added index on favorite.added_at for faster sorting
  • Added person_count column to database_info table

🔧 Improvements

Service Layer Updates

• database.service.ts - Queries from SQLite with JSON fallback
• search.service.ts - Uses FTS5 for text search, supports cross-database global search
• path.service.ts - Recursive CTEs for shortest/longest/random paths
• favorites.service.ts - SQLite storage with JSON backup
• augmentation.service.ts - External identity registration

Schema Design

• Normalized tables: person, external_identity, parent_edge, spouse_edge
• Extensible claims system for facts with provenance
• Vital events with date parsing (supports BC notation)
• Database membership for multi-tree support
• Full schema in server/src/db/schema.sql

Developer Experience

• Updated CLAUDE.md with comprehensive migration documentation
• Added npm scripts for migration management
• Better error messages for ID resolution failures

📦 Installation

git clone https://github.com/atomantic/SparseTree.git
cd SparseTree
npm run install:all
npm run migrate
pm2 start ecosystem.config.cjs

🔄 Upgrading from v0.2.x

Run the update script to automatically migrate:

./update.sh

Or manually:

git pull origin main
npm install
npm run build
npm run migrate
pm2 restart ecosystem.config.cjs

🔗 Full Changelog

Full Diff: v0.2.11...v0.3.21

v0.2.10

Release v0.2.10 - Multi-Platform Linking & Tree Views

Released: YYYY-MM-DD

Overview

Major enhancements to the ancestry tree visualization with four view modes and lazy loading for deep ancestry. Added multi-platform genealogy linking with Ancestry.com and WikiTree integration.

🎉 New Features

Multiple Tree View Modes

• Focus Navigator: Navigate one person at a time with breadcrumb trail
• Pedigree Chart: Classic vertical tree with expandable generations
• Generational Columns: Horizontal columns organized by generation
• Classic: Original SVG-based tree with zoom/pan

Lazy Loading for Deep Ancestry

• Initial load of 10 generations for columns view
• "Load 5 more" button appears when more ancestors are available
• Successfully tested loading 21 generations with 17,000+ ancestors

View Mode URL Persistence

• Tree view mode saved in URL query params (?view=columns)

Ancestry.com Linking

• Link persons to their Ancestry.com profiles with automatic photo extraction
• Browser-based scraping with auto-login support using saved credentials
• Srcset parsing to extract highest resolution photos (5x = maxside=1800)
• Auto-launches/connects browser when needed

WikiTree Linking

• Link persons to their WikiTree profiles with photo extraction
• HTTP-based scraping (no auth required for public profiles)
• Extracts profile photo and description

Manual Photo Selection

• "Use Photo" button for each linked platform
• Separated linking from photo fetching for user control
• Support for FamilySearch, Wikipedia, Ancestry, and WikiTree photos
• Photos stored locally with platform-specific naming ({personId}-ancestry.jpg, etc.)

Unified Platforms UI

• Consolidated all platform links into single "Platforms" section in PersonDetail
• FamilySearch, Wikipedia, Ancestry, WikiTree all shown together
• Link and "Use Photo" buttons for each platform

Sample Database

• Included sample genealogy database (G849-MHS) for new users to explore the app
• Medieval Swiss nobility lineage (920-968 AD) with 4 persons
• Sample databases load automatically and are marked with isSample flag
• Cannot be deleted (protected from accidental removal)

🔧 Improvements

Genealogy Providers

• All 8 supported genealogy providers (FamilySearch, Ancestry, WikiTree, MyHeritage, Geni, FindMyPast, Find A Grave, 23andMe) are now pre-populated on first launch
• No more manually adding each provider - just enable and configure the ones you want to use

Tree Visualization

• Simplified Generation Labels: Gen 3+ shows "1st/2nd/3rd Great-Grandparents" instead of verbose labels
• Columns View Optimization: Only displays known ancestors (hides unknown placeholders)
• Header Styling: Fixed transparent header in columns view to prevent text collision when scrolling

Photo Priority

• Updated sparse tree view to use photos in order:
  1. Ancestry (highest priority)
  2. WikiTree
  3. Wikipedia
  4. FamilySearch scraped (lowest priority)

Data Model

• Added photoUrl field to PlatformReference to store discovered photo URLs before downloading
• Added support for structured name fields:
  • birthName: Birth/maiden name when display name differs (e.g., married name is preferred)
  • marriedNames: Array of names taken after marriage
  • aliases: Array of "also known as" names (nicknames, alternate spellings)
  • Maintains backwards-compatible alternateNames array

🐛 Bug Fixes

• Browser auto-connects when linking Ancestry profiles (no more "Browser not connected" errors)
• Ancestry photo now appears in sparse tree view
• Indexer now handles network timeouts gracefully with retry logic instead of crashing
  • ETIMEDOUT, ECONNRESET, and other transient errors automatically retry up to 3 times
  • Exponential backoff (5s, 10s, 20s) between retries
  • Failed fetches skip the person and continue indexing instead of exiting

📦 Installation

git clone https://github.com/atomantic/SparseTree.git
cd SparseTree
npm run install:all
pm2 start ecosystem.config.cjs

🔗 Full Changelog

Full Diff: v0.1.x...v0.2.10

v0.1.0

Release v0.1.0

Features

• Local web UI enhancement of FamilySearchFinder
• Multi-platform genealogy provider support (FamilySearch, Ancestry, WikiTree, 23andMe)
• Browser-based scraping with Playwright automation
• Provider login credentials with encrypted storage for auto-authentication
• FamilySearch-style ancestry tree visualization
• Favorites system with sparse tree visualization
• GEDCOM import/export support
• Path finding between ancestors (shortest/longest/random)
• Light/dark theme support

Installation

git clone https://github.com/atomantic/SparseTree.git
cd SparseTree
npm run install:all
pm2 start ecosystem.config.cjs