feat(ups): implement queue subs

[MasterPtato]

Description

Please include a summary of the changes and the related issue. Please also include relevant motivation and context.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[railway-app]

🚅 Deployed to the rivet-pr-4486 environment in rivet-frontend

Service	Status	Web	Updated (UTC)
kitchen-sink	❌ Build Failed (View Logs)	Web	Apr 5, 2026 at 11:23 am
website	😴 Sleeping (View Logs)	Web	Apr 5, 2026 at 12:09 am
frontend-cloud	😴 Sleeping (View Logs)	Web	Apr 2, 2026 at 4:26 am
frontend-inspector	😴 Sleeping (View Logs)	Web	Mar 24, 2026 at 7:28 am
mcp-hub	✅ Success (View Logs)	Web	Mar 24, 2026 at 12:31 am
ladle	❌ Build Failed (View Logs)	Web	Mar 24, 2026 at 12:31 am

[MasterPtato]

• chore: envoy client #4521 : 2 dependent PRs (#4531 , #4534 )
• feat(pb): actors v3 #4463
• feat(ups): implement queue subs #4486 👈 (View in Graphite)
• feat(cache): add in flight deduping #4459
• fix(cache): clean up lib #4457
• fix(config): allow configuring topo dcs via map, fix pg ssl mode config #4456
• fix(api): dont 500 on cross dc response #4449
• chore: update byte units on frontend #4444
• fix: runner alloc idx logic, api auth for actor get #4443
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[pkg-pr-new]

More templates

• actor-actions
• actor-actions-vercel
• ai-agent
• ai-agent-vercel
• example-ai-and-user-generated-actors-freestyle
• chat-room
• chat-room-vercel
• example-cloudflare-workers
• example-cloudflare-workers-hono
• collaborative-document
• collaborative-document-vercel
• cross-actor-actions
• cross-actor-actions-vercel
• example-cursors
• example-cursors-raw-websocket
• example-cursors-raw-websocket-vercel
• example-cursors-vercel
• custom-serverless
• example-elysia
• example-experimental-durable-streams-ai-agent
• example-experimental-durable-streams-ai-agent-vercel
• geo-distributed-database
• geo-distributed-database-vercel
• hello-world
• hello-world-vercel
• example-hono
• hono-react
• hono-react-vercel
• example-kitchen-sink
• example-kitchen-sink-vercel
• multi-region-vercel
• multiplayer-game
• example-multiplayer-game-patterns
• example-multiplayer-game-patterns-vercel
• multiplayer-game-vercel
• native-websockets
• native-websockets-vercel
• next-js
• per-tenant-database
• per-tenant-database-vercel
• example-raw-fetch-handler
• example-raw-fetch-handler-vercel
• example-raw-websocket-handler
• example-raw-websocket-handler-proxy
• example-raw-websocket-handler-proxy-vercel
• example-raw-websocket-handler-vercel
• example-react
• example-react-vercel
• sandbox
• sandbox-coding-agent
• sandbox-coding-agent-vercel
• sandbox-vercel
• scheduling
• scheduling-vercel
• example-sqlite-drizzle
• example-sqlite-raw
• state
• state-vercel
• example-stream
• example-stream-vercel
• example-trpc

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@4486

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@4486

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@4486

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@4486

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@4486

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@4486

@rivetkit/sqlite-vfs

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sqlite-vfs@4486

@rivetkit/traces

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/traces@4486

@rivetkit/workflow-engine

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/workflow-engine@4486

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@4486

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@4486

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@4486

commit: ec32bdf

[claude]

PR Review: feat(ups): implement queue subs

The core design is solid. Using FOR UPDATE SKIP LOCKED for Postgres-backed competing consumers is the right approach, and the heartbeat/TTL pattern for publisher-side discovery is appropriate given Postgres NOTIFY's lack of built-in queue semantics. A few issues need attention before merging.

Critical

DefaultHasher is not stable across processes or Rust versions
(engine/packages/universalpubsub/src/driver/postgres/mod.rs)

DefaultHasher's output is explicitly not guaranteed to be deterministic across processes or Rust versions. Using it to compute subject_hash and queue_hash stored in ups_queue_subs / ups_queue_messages means two processes, or the same process after a Rust upgrade, may compute different hashes for the same string, silently breaking all queue routing. The existing hash_subject path has the same pre-existing bug, but extending it to persisted queue keys makes this a data-integrity issue.

Fix: Replace with a stable, deterministic hash (e.g., FNV-1a or truncated SHA-256 formatted as hex).

Medium

No metrics update in queue subscription cleanup task (spawn_queue_subscription_cleanup_task)

spawn_subscription_cleanup_task updates metrics::POSTGRES_SUBSCRIPTION_COUNT when removing entries. The queue analog does not update any counter after removing from queue_subscriptions. This is an observability gap.

Unresolved TODO: retain_sync inside retain_async (engine/packages/universalpubsub/src/driver/memory/mod.rs)

The TODO comment is left in the code. retain_sync inside the retain_async closure blocks the async executor thread for the duration of the inner lock. For a 60-second GC interval this is acceptable (no deadlock risk since the nested maps are distinct objects), but the TODO should be replaced with a comment explaining why it is safe.

No queue depth limit (engine/packages/universalpubsub/src/driver/postgres/mod.rs)

QUEUE_MESSAGE_MAX_AGE_SECS is 3600s and GC runs every 5 minutes. If a consumer is down, messages accumulate unbounded for up to an hour. High-throughput queues with dead consumers could grow the table very large. Worth noting as a design-level concern even if not a blocker.

Low

Idiomatic Rust: return Ok(...) vs expression form (engine/packages/universalpubsub/src/pubsub.rs:136)

return Ok(Subscriber::new(...)) should be the bare expression Ok(Subscriber::new(...)) to match the style of the surrounding subscribe method.

SQL injection pattern is fragile (engine/packages/universalpubsub/src/driver/postgres/mod.rs)

NOTIFY "{}" with channel built from format! is safe today only because the hash output is hex. The same pre-existing pattern is now copied into two additional call sites. A helper that validates the channel name before interpolating would prevent this from becoming a vulnerability if the input ever changes.

Unused anyhow! import (engine/packages/universalpubsub/src/driver/postgres/mod.rs:1)

If all usages of anyhow! were replaced by .context() per the project convention, remove anyhow from the import line.

Informational

• Memory metric counts topics, not subscribers. MEMORY_SUBSCRIBER_COUNT uses self.subscribers.len() + self.queue_subscribers.len() which counts distinct topic entries, not individual sender handles. Pre-existing issue, but the surface is now expanded.
• Test timing sensitivity. The load-balance test uses a 500ms collection window per subscriber (vs. 5s in test_queue_subscribe_single). This may be flaky on slow CI/Postgres environments with 10 messages across database round-trips.
• NATS driver is minimal and correct - clean delegation to the NATS client.

Summary

Severity	Issue
Critical	DefaultHasher used for persistent DB keys - not stable across processes/versions
Medium	No metrics update in queue subscription cleanup task
Medium	retain_sync inside async retain - TODO unresolved
Medium	No queue depth limit; unbounded growth if consumers are down
Low	return Ok(...) vs idiomatic expression form
Low	SQL NOTIFY injection pattern fragile (safe today, not future-proof)
Low	Unused anyhow! import if all usages replaced
Info	Memory metric counts topics not subscribers (expanded surface)
Info	Load-balance test 500ms window may be flaky on slow CI

The DefaultHasher stability issue is the most important fix - it will silently corrupt queue routing in any multi-process deployment.

[claude]

Code Review summary for feat(ups): implement queue subs -- The implementation is well-structured and the test coverage is solid. Two medium issues to address before merge: (1) retain_sync on scc::HashMap inside the memory driver GC task runs synchronously inside an async closure and can block the executor; (2) The SELECT and INSERT+NOTIFY in publish_to_queues are not wrapped in a transaction, creating a race where a message can be orphaned for up to 1 hour if the subscriber dies between the two operations. Minor items: one remaining anyhow! macro call in publish() should use .context() instead; the queue parameter is missing from the tracing span in queue_subscribe (fields(%subject) should also include queue); and the Cargo.lock version downgrade from 2.1.7 to 2.1.6-rc.1 appears to be a stacked-PR artifact. On test coverage: test_queue_subscribe_load_balance only checks total message count, not per-subscriber distribution -- this is the right choice to avoid flakiness but worth noting in a comment. Also missing a test for Postgres reconnection behavior (does not drain pending ups_queue_messages on reconnect).

[claude]

PR Review: feat(ups): implement queue subs (#4486)

Good implementation of queue subscription (queue group) semantics across all three drivers. The overall design is solid — NATS delegates to the native implementation, memory uses random selection, and Postgres uses a durable table-backed approach with heartbeats. The three integration tests cover the key correctness properties.

A few issues worth noting:

---

Bug: Heartbeat task leaks on subscriber drop

engine/packages/universalpubsub/src/driver/postgres/mod.rs

let heartbeat_token = tokio_util::sync::CancellationToken::new();
let heartbeat_token_child = heartbeat_token.clone();
// ...
_heartbeat_token: heartbeat_token,

Dropping a CancellationToken does not cancel it — only calling .cancel() does. When PostgresQueueSubscriber is dropped, the _heartbeat_token field is dropped but the heartbeat task holds heartbeat_token_child and keeps running indefinitely. The Drop impl deletes the DB row, so the heartbeat's UPDATE silently no-ops, but the task and pool connections are wasted every 10 seconds.

Fix: use drop_guard() so cancel fires on drop:

let heartbeat_token = tokio_util::sync::CancellationToken::new();
let heartbeat_token_child = heartbeat_token.clone();
let _heartbeat_guard = heartbeat_token.drop_guard(); // cancel on drop
// store _heartbeat_guard on the struct instead of heartbeat_token

---

Test flakiness risk in test_queue_subscribe_load_balance

tokio::time::timeout(Duration::from_millis(500), sub1.next())

Using a 500ms timeout to collect messages is inherently flaky under slow CI or Postgres startup. A more robust approach is to collect exactly N messages with a longer per-message timeout:

for _ in 0..message_count {
    match tokio::time::timeout(Duration::from_secs(5), /* ... */).await { ... }
}

---

GC task not cancellable

The GC task spawned in new() has no shutdown mechanism and will run forever even after all references to the driver are dropped. This is a minor resource leak. Consider tying it to a CancellationToken stored in the driver (or using tokio_util::task::TaskTracker).

---

Memory driver metrics count topics, not subscribers

metrics::MEMORY_SUBSCRIBER_COUNT
    .set((inner.subscribers.len() + inner.queue_subscribers.len()) as i64);

Both .len() calls count distinct topics (outer map entries), not the total number of individual subscriber channels. This was already the case for regular subs, but worth tracking. The queue variant additionally undercounts since multiple queue groups per topic are flattened.

---

Minor: unnecessary return in pubsub.rs

return Ok(Subscriber::new(...));

The explicit return is not idiomatic Rust; just Ok(Subscriber::new(...)) is cleaner.

---

Noted — by design

• Sequential insert+notify per queue group in publish_to_queues: Each group gets its own INSERT + NOTIFY in a loop, not a transaction. A partial failure leaves some groups notified and others not. Given the heartbeat-based fallback and GC, this is acceptable but worth a comment.
• FOR UPDATE SKIP LOCKED CTE: Clean approach for at-most-once delivery. The Lagged broadcast error handling correctly loops back to claim, so no messages are silently dropped.
• Reconnection support: Queue channels are correctly re-LISTENed alongside regular channels after reconnect.

Overall solid work. The heartbeat leak is the main issue worth a follow-up fix.

[NathanFlurry]

Merge activity

• Apr 5, 11:11 AM UTC: A user started a stack merge that includes this pull request via Graphite.
• Apr 5, 11:23 AM UTC: Graphite rebased this pull request as part of a merge.
• Apr 5, 11:23 AM UTC: @NathanFlurry merged this pull request with Graphite.