Refactor domain scheduling and management

[jakejarvis]

Remove incremental PPR, adjust cron schedules, add last accessed tracking, and implement dead letter queue for failed tasks. Enhance database schema with new fields for change frequency and last accessed timestamp.

Summary by CodeRabbit

• New Features

  • Priority-based scheduling system ensuring high-priority domains are processed first
  • Dead-letter queue management for automatic recovery from failed revalidations
  • Adaptive TTL adjustment based on domain change frequency

• Bug Fixes

  • Staleness checks to prevent revalidating outdated domains
  • Automatic cleanup of orphaned queue entries

• Tests

  • Added comprehensive test coverage for queue management and priority handling

• Chores

  • Adjusted cron schedules for improved performance
  • Enhanced failure tracking and retry mechanisms
  • Disabled partial pre-rendering feature

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
domainstack	Ready	Preview	Comment	Oct 29, 2025 9:16pm

[coderabbitai]

Walkthrough

Adds domain access tracking with last-accessed timestamps, introduces a priority-based scheduling system for section revalidation with dead-letter queue (DLQ) support, implements adaptive TTL calculations, and adds a cron job for database cleanup. Includes database migrations, new queue management capabilities, and enhanced scheduling logic.

Changes

Cohort / File(s)	Summary
Database Schema Updates drizzle/0004_flat_masque.sql, drizzle/meta/0004_snapshot.json, drizzle/meta/_journal.json, lib/db/schema.ts	Adds lastAccessedAt and changeFrequency columns to domains table with corresponding migration files, snapshot, and journal entries.
Domain Access & TTL Management lib/db/repos/domains.ts, lib/db/ttl.ts, app/[domain]/page.tsx	Adds updateLastAccessed, incrementChangeFrequency, and getChangeFrequency functions to domains repository. Introduces adaptiveTtl function for dynamic TTL adjustment. Integrates last-access tracking in domain page. Removes experimental_ppr export.
Scheduling & Queue System lib/schedule.ts, lib/schedule.test.ts, lib/constants.ts	Implements priority lanes (high/normal/low), dead-letter queue handling with moveToDLQ/restoreFromDLQ, failure tracking with getFailureAttempts, and orphaned entry cleanup. Expands test coverage for DLQ, priority scheduling, and cleanup logic. Updates cron timing and adds failure thresholds.
Cron Infrastructure app/api/cron/queue-cleanup/route.ts, vercel.json	Adds GET handler for queue cleanup cron job with authorization verification. Updates due-drain schedule from 2 to 10 minutes and adds queue-cleanup cron for Sundays at midnight.
Configuration & Services next.config.ts, server/services/registration.ts, lib/redis-mock.ts	Removes experimental PPR configuration. Makes background revalidation scheduling conditional on isRegistered flag. Adds hgetall and pipeline methods to Redis mock for bulk operations and transaction support.

Sequence Diagram(s)

sequenceDiagram
    participant User as User Request
    participant Page as app/[domain]/page
    participant Repo as Domains Repo
    participant Schedule as Schedule System
    participant DLQ as Dead-Letter Queue
    participant Cron as Queue-Cleanup Cron

    User->>Page: Load domain page
    Page->>Repo: updateLastAccessed(domain)
    Repo->>Repo: Update lastAccessedAt
    
    Note over Schedule: Scheduled Revalidation
    Schedule->>Schedule: Acquire lock + Revalidate
    
    alt Success
        Schedule->>Repo: incrementChangeFrequency()
        Repo->>Repo: Update changeFrequency++
        Schedule->>Schedule: Reset backoff, write result
    else Failure
        Schedule->>Schedule: recordFailureAndBackoff()
        Schedule->>Schedule: Increment failure attempts
        alt Max attempts reached
            Schedule->>DLQ: moveToDLQ(section, domain)
            DLQ->>DLQ: Store in DLQ queue<br/>with cooldown
        end
    end
    
    Schedule->>Schedule: Release lock (all paths)
    
    Note over Cron: Periodic Cleanup
    Cron->>Schedule: cleanupOrphanedQueueEntries()
    Schedule->>Schedule: Cross-check Redis<br/>vs Database
    Schedule->>Schedule: Remove orphaned entries

Loading

Poem

🐰 Domain tracks its access time so bright,
Priority lanes bring order to the night,
When tasks keep failing, DLQ's the way,
Dead letters rest, then retry another day!
Cleanup cron sweeps orphans away,
Adaptive TTL wins the fray! ✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The pull request title "Refactor domain scheduling and management" is directly related to the actual changes in the PR. The changeset substantially modifies domain scheduling logic by implementing a dead-letter queue system, adding priority-based scheduling, enabling staleness-based validation, and enhancing the scheduling system with failure tracking and adaptive TTL calculations. The title also aligns with management improvements like last-accessed tracking and orphaned entry cleanup. While the title could be more specific in highlighting key features like DLQ and priority lanes, it accurately identifies the primary areas of change and would give a teammate scanning the commit history a meaningful understanding that domain scheduling and management systems were refactored.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch fix/inngest-quota-take-2

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: ASSERTIVE

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between c0634ae and 1970afc.

📒 Files selected for processing (16)

• app/[domain]/page.tsx (2 hunks)
• app/api/cron/queue-cleanup/route.ts (1 hunks)
• drizzle/0004_flat_masque.sql (1 hunks)
• drizzle/meta/0004_snapshot.json (1 hunks)
• drizzle/meta/_journal.json (1 hunks)
• lib/constants.ts (1 hunks)
• lib/db/repos/domains.ts (2 hunks)
• lib/db/schema.ts (1 hunks)
• lib/db/ttl.ts (1 hunks)
• lib/inngest/functions/section-revalidate.ts (3 hunks)
• lib/redis-mock.ts (2 hunks)
• lib/schedule.test.ts (2 hunks)
• lib/schedule.ts (8 hunks)
• next.config.ts (0 hunks)
• server/services/registration.ts (1 hunks)
• vercel.json (2 hunks)

💤 Files with no reviewable changes (1)

• next.config.ts

🧰 Additional context used

🧬 Code graph analysis (7)

app/[domain]/page.tsx (1)

lib/db/repos/domains.ts (1)

• updateLastAccessed (52-65)

app/api/cron/queue-cleanup/route.ts (1)

lib/schedule.ts (1)

• cleanupOrphanedQueueEntries (284-362)

server/services/registration.ts (1)

lib/schedule.ts (1)

• scheduleSectionIfEarlier (119-137)

lib/schedule.test.ts (5)

lib/schedule.ts (6)

• moveToDLQ (194-224)
• restoreFromDLQ (230-254)
• getFailureAttempts (259-270)
• scheduleSectionIfEarlier (119-137)
• drainDueDomainsOnce (376-497)
• cleanupOrphanedQueueEntries (284-362)

lib/redis.ts (2)

• redis (4-4)
• ns (9-11)

lib/constants.ts (2)

• MAX_FAILURE_ATTEMPTS (85-85)
• MAX_EVENTS_PER_RUN (73-73)

lib/db/schema.ts (1)

• domains (70-91)

lib/redis-mock.ts (1)

• resetInMemoryRedis (27-29)

lib/inngest/functions/section-revalidate.ts (4)

lib/cache.ts (1)

• acquireLockOrWaitForResult (26-114)

lib/redis.ts (1)

• redis (4-4)

lib/schedule.ts (4)

• resetFailureBackoff (180-188)
• recordFailureAndBackoff (163-178)
• getFailureAttempts (259-270)
• moveToDLQ (194-224)

lib/constants.ts (1)

• MAX_FAILURE_ATTEMPTS (85-85)

lib/schedule.ts (5)

lib/redis.ts (2)

• ns (9-11)
• redis (4-4)

lib/constants.ts (2)

• DLQ_COOLDOWN_HOURS (86-86)
• STALE_ACCESS_THRESHOLD_DAYS (82-82)

lib/db/client.ts (1)

• db (24-24)

lib/db/schema.ts (1)

• domains (70-91)

lib/db/repos/domains.ts (1)

• findDomainByName (39-46)

lib/db/repos/domains.ts (2)

lib/db/client.ts (1)

• db (24-24)

lib/db/schema.ts (1)

• domains (70-91)

🔇 Additional comments (11)

drizzle/0004_flat_masque.sql (1)

1-3: LGTM! Clean migration with appropriate defaults.

The migration safely adds tracking fields for domain access patterns and change frequency. The nullable last_accessed_at ensures backward compatibility with existing domains, and the index on this field will support efficient staleness queries during cleanup operations.

server/services/registration.ts (1)

327-341: LGTM! Sensible optimization to skip scheduling for unregistered domains.

The conditional scheduling prevents unnecessary background tasks for domains that have no registration data to revalidate. Error handling remains appropriate with non-blocking warnings.

drizzle/meta/_journal.json (1)

33-39: LGTM! Standard Drizzle migration journal entry.

vercel.json (2)

11-12: LGTM! Cron interval adjustment aligns with updated constants.

The drain interval change from every 2 minutes to every 10 minutes reduces system load and aligns with DRAIN_CRON_MINUTES = 10 in lib/constants.ts.

---

22-25: LGTM! Appropriate weekly schedule for cleanup tasks.

The weekly cleanup cron (Sunday midnight) is suitable for non-time-critical orphan removal operations.

app/api/cron/queue-cleanup/route.ts (1)

1-51: LGTM! Well-structured cron handler with proper authentication and error handling.

The implementation correctly:

• Validates the CRON_SECRET before executing cleanup
• Measures and logs operation duration
• Returns structured responses for both success and failure cases
• Uses force-dynamic to ensure fresh execution on each invocation

lib/inngest/functions/section-revalidate.ts (2)

85-121: LGTM! Clean lock-acquisition pattern with appropriate skip logic.

The two-step approach correctly:

• Attempts to acquire the lock or wait for results from another process
• Skips processing when the lock cannot be acquired (avoiding duplicate work)
• Returns structured results for downstream cleanup decisions

---

126-185: LGTM! Comprehensive cleanup with DLQ support.

The cleanup phase properly handles:

• Writing deduplication results on success
• Resetting failure backoff after successful revalidation
• Recording failures with exponential backoff
• Moving domains to DLQ after reaching MAX_FAILURE_ATTEMPTS
• Guaranteeing lock release via finally block

The structured error handling at each step ensures resilience.

lib/redis-mock.ts (2)

246-250: LGTM! Clean implementation of hgetall.

Returns an empty object for non-existent or empty hashes, matching Redis behavior.

---

252-300: LGTM! Well-designed pipeline implementation.

The chainable API correctly:

• Queues commands for batch execution
• Returns error-result tuples for each command
• Continues execution even if a command fails (matches Redis pipeline semantics)

app/[domain]/page.tsx (1)

50-51: LGTM! Appropriate fire-and-forget pattern for access tracking.

The updateLastAccessed call correctly:

• Runs after domain validation and canonicalization
• Uses fire-and-forget to avoid blocking page rendering
• Swallows errors silently (consistent with non-critical tracking pattern used elsewhere)

The implementation in lib/db/repos/domains.ts already includes debug logging for failures, so the empty catch here is acceptable.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

❌ Patch coverage is 58.28571% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.92%. Comparing base (c0634ae) to head (1970afc).
⚠️ Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
lib/schedule.ts	68.59%	29 Missing and 9 partials ⚠️
lib/redis-mock.ts	44.44%	14 Missing and 1 partial ⚠️
lib/db/repos/domains.ts	0.00%	12 Missing ⚠️
lib/db/ttl.ts	0.00%	5 Missing ⚠️
server/services/registration.ts	40.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #152      +/-   ##
==========================================
- Coverage   69.63%   68.92%   -0.72%     
==========================================
  Files         101      102       +1     
  Lines        2747     2909     +162     
  Branches      852      885      +33     
==========================================
+ Hits         1913     2005      +92     
- Misses        514      574      +60     
- Partials      320      330      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jakejarvis]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.