ConversationStore: Implement migration to postgres #4810

akshaymankar · 2025-10-09T15:08:13Z

https://wearezeta.atlassian.net/browse/WPB-20751

Checklist

Add a new entry in an appropriate subdirectory of changelog.d
Read and follow the PR guidelines

…tion

… during migration Exceptions: 1. Getting a paginated list of qualified conv Ids 2. It will not work well if the migration fails to delete conv data from Cassandra after copying it to postgres These problems will be solved in following commits

When fetching `maxIds + 1` convs, it can happen that a user has exactly those many convs left, in this case the `hasMore` field of the page would be false, but we'd be sending a truncated list of convs.

This is done by making the pagingState encode last conversation Id served. The store effect has a new action to list only remote conv ids. The `GetConverastionIds` action has been removed and implemented generally using `GetLocalConverastionIds` and `GetRemoteConversationIds`. This makes `MultiTabelPage` type obsolete for conv ids, but its still kept around so we don't break any APIs.

…nv in postgres This is consistent with creating new conversations in postgres. This way when the migration is complete already running galley instances won't create more data in Cassandra

…tatuses of a user

…ndra are never read Exception: Listing local conversation ids and listing team conversation ids

…nvs to postgres

…d worker

…f a user

Copilot

Pull Request Overview

This PR implements support for migrating conversation data from Cassandra to PostgreSQL, enabling channel search and management in team UI. The migration occurs in three phases: preparation (write to both DBs), active migration (background worker moves existing data), and PostgreSQL-only operation.

Key changes:

Adds a new MigrationToPostgresql storage location option for phased migration
Implements background worker to migrate conversations and remote member data
Introduces migration locks to prevent race conditions during concurrent operations

Reviewed Changes

Copilot reviewed 43 out of 43 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
services/galley/src/Galley/Options.hs	Adds new MigrationToPostgresql storage option with documentation
services/galley/src/Galley/App.hs	Adds polysemy-conc dependency and MigrationError handling
libs/wire-subsystems/src/Wire/ConversationStore/*.hs	Implements dual-write interpreter and migration logic
services/background-worker/src/Wire/*.hs	Adds migration worker with Prometheus metrics
charts/background-worker/*.yaml	Adds Cassandra Galley connection and PostgreSQL config
integration/test/Test/Conversation/Migration.hs	Comprehensive migration tests for MLS and Proteus conversations

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

services/galley/src/Galley/Options.hs

services/background-worker/src/Wire/MigrateConversations.hs

libs/wire-subsystems/src/Wire/ConversationStore/Migration/Cleanup.hs

charts/background-worker/values.yaml

services/background-worker/src/Wire/BackgroundWorker.hs

libs/wire-subsystems/src/Wire/ConversationStore/Postgres.hs

libs/wire-subsystems/src/Wire/ConversationStore/Cassandra.hs

libs/wire-subsystems/src/Wire/ConversationStore/Migration.hs

Copilot

Pull Request Overview

Copilot reviewed 43 out of 43 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

services/background-worker/src/Wire/MigrateConversations.hs

changelog.d/0-release-notes/conversation-migration

integration/test/Test/Conversation/Migration.hs

libs/wire-subsystems/src/Wire/ConversationStore/Cassandra.hs

libs/wire-subsystems/src/Wire/ConversationStore/Migration.hs

pcapriotti

Looks good.

integration/test/Test/Conversation/Migration.hs

pcapriotti · 2025-10-29T14:20:26Z

.../wire-subsystems/postgres-migrations/20251015124151-create-conversation-migration-status.sql

+CREATE TABLE conversation_migration_pending_deletes (
+  typ text NOT NULL,
+  id uuid NOT NULL,
+  PRIMARY KEY (typ, id)


Why no foreign key constraint here?

This is to track pending deletes in Cassandra, in case the conv gets deleted from Postgres while its pending in cassandra we don't want to also forget to delete it from Cassandra. So I think it necessary to not create a foreign key, because that will either block conv from being deleted in postgres or if we do a cascading delete, make us forget that this conv needs to be deleted in Cassandra and the migration code will revive this conv.

Makes sense.

libs/wire-subsystems/src/Wire/ConversationStore/Postgres.hs

…und-worker

Because sometimes we don't have to get an empty page to realize we're at the end

Also stop any throttling, OOMKilling

…n faster

… in same order as IDs

akshaymankar force-pushed the conversation-postgres-ci branch from c32b073 to 4e1c43c Compare October 13, 2025 09:39

akshaymankar force-pushed the conversation-migration branch from 905ece0 to ea4a8bd Compare October 13, 2025 09:39

zebot added the ok-to-test Approved for running tests in CI, overrides not-ok-to-test if both labels exist label Oct 13, 2025

akshaymankar force-pushed the conversation-postgres-ci branch from 4e1c43c to 619999d Compare October 13, 2025 11:40

akshaymankar force-pushed the conversation-migration branch from ea4a8bd to ce9435e Compare October 13, 2025 11:42

Base automatically changed from conversation-postgres-ci to develop October 13, 2025 13:36

akshaymankar force-pushed the conversation-migration branch 2 times, most recently from 2b79ef3 to 5090c28 Compare October 15, 2025 08:35

akshaymankar force-pushed the conversation-migration branch 2 times, most recently from 4ca15c9 to e44c50d Compare October 28, 2025 12:19

akshaymankar added 20 commits October 29, 2025 09:08

ConversationStore.Migration: Implement function to migrate a conversa…

835e0fb

…tion

ConverastionStore.Cassandra: Fix edge case bug in getLocalConvIds

1a736f9

When fetching `maxIds + 1` convs, it can happen that a user has exactly those many convs left, in this case the `hasMore` field of the page would be false, but we'd be sending a truncated list of convs.

ConversationStore.Cassandra: Save users joining their first remote co…

8372a09

…nv in postgres This is consistent with creating new conversations in postgres. This way when the migration is complete already running galley instances won't create more data in Cassandra

ConversationStore.Cassandra: Use embedClient

b269aa7

ConversationStore.Migration: Delete obsolete stub

2eae57c

ConversationStore.Migration: Implement function to migrate a remote s…

970c6ff

…tatuses of a user

ConversationStore.Migration: Add top level functions to do the migration

119c5ef

Conversation.{Migration,Cassandra}: Ensure pending deletes from Cassa…

683e113

…ndra are never read Exception: Listing local conversation ids and listing team conversation ids

ConverastionStore.Cassandra: Document limitation in listing conv ids

db927ba

galley: Allow choosing migration interpreter for ConversationStore

810a339

background-worker: Integrate with ConvSubsystem to allow migrating co…

bb6d7bc

…nvs to postgres

integration: Set cassandra keyspace and pg db names for dyn backgroun…

c06d397

…d worker

background-worker: Fix tests

cf76768

ConversationStore.Cassandra: Fix queries for selecting remote convs o…

524b75d

…f a user

integration: Add basic test to test conversation migration

a50ae6d

background-worker: Fix copy-pasta error with cassandra galley

fca56af

background-worker: Add some observability to pg migration

7089959

integration: Use background worker metrics to check migration completion

9d7fb56

Copilot AI reviewed Oct 29, 2025

View reviewed changes

Fix typos

1e7b7dc

akshaymankar requested a review from Copilot October 29, 2025 15:48

Copilot AI reviewed Oct 29, 2025

View reviewed changes

blackheaven approved these changes Oct 29, 2025

View reviewed changes

libs/wire-subsystems/src/Wire/ConversationStore/Migration.hs Show resolved Hide resolved

libs/wire-subsystems/src/Wire/ConversationStore/Migration.hs Show resolved Hide resolved

pcapriotti reviewed Oct 30, 2025

View reviewed changes

akshaymankar added 6 commits October 30, 2025 10:14

Fix more typos

b078f34

ConversationStore.Migration: Declare type of insertConv

2a05314

Galley.Options: Fix error message

58ed0ed

integration-setup: Configure cassandraGalley and postgres for backgro…

972168b

…und-worker

integration-setup: Correct set max_connections for postgresql

eef622b

galley-integration: Change assertions about pagination

3b51ab0

Because sometimes we don't have to get an empty page to realize we're at the end

akshaymankar force-pushed the conversation-migration branch from 13322d3 to ca61c3c Compare October 30, 2025 14:57

integration-setup: More CPU/memory for postgresql

192ce20

Also stop any throttling, OOMKilling

akshaymankar force-pushed the conversation-migration branch from ca61c3c to 192ce20 Compare October 30, 2025 15:19

integration-setup: Allow even more connections to postgresql

0cb0040

akshaymankar force-pushed the conversation-migration branch from e775514 to e08cf96 Compare November 4, 2025 09:47

akshaymankar added 3 commits November 4, 2025 11:04

ConversationStore.Migration: Add name of the migration to all logs

8e886a7

charts/integration: Mount background-worker secrets

fb5cad7

integration/Test.Conv.Migration: Create fewer other convs so tests ru…

0f3c1a2

…n faster

akshaymankar force-pushed the conversation-migration branch from 378682d to 0f3c1a2 Compare November 4, 2025 10:04

akshaymankar added 5 commits November 4, 2025 15:34

integration: Set cassandra keyspace for dynamic cannons

3955081

integration-setup: Allow tests to run for 5 more mins

c8a2a55

galley-integration: Produce better error message

699cad2

docs: Add steps for migrations

7a10226

galley-integration: Relax the requirement that getConvs returns convs…

4a31f48

… in same order as IDs

akshaymankar merged commit cc06340 into develop Nov 5, 2025
10 checks passed

akshaymankar deleted the conversation-migration branch November 5, 2025 12:32

This was referenced Nov 26, 2025

Release 2025-11-26 - (expected chart version 5.24.0) #4875

Closed

Release 2025-11-26 - (expected chart version 5.24.0) #4876

Merged

ConversationStore: Implement migration to postgres #4810

ConversationStore: Implement migration to postgres #4810

Uh oh!

Conversation

akshaymankar commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pcapriotti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pcapriotti Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

akshaymankar Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

pcapriotti Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

akshaymankar commented Oct 9, 2025 •

edited

Loading