-
Notifications
You must be signed in to change notification settings - Fork 8
Description
error handling todos:
invalid/failing message
depends on the reason for failing
if error message is due to invalidity (eg: upsert for a column that does not exist), we should delete it
consideration: we process messages in batched DB calls, so if there is just 1 invalid message in a batch, ideally the other messages should succeed and we should get a clear error pointing to the exact message(s) that failed so that we can then delete those messages. it's unclear if this exact functionality can be achieved through plain existing db queries (eg: some kind of conflict/error handling within the postgres query, and some kind of comprehensive, high fidelity error response). given this, we should investigate the scenario and improve our query to provide this, or if not possible, find an alternative.
for now, since all messages are in our control, we'll detect this quickly if a mistake is made and be able to remedy it before deploying to production, but as soon as we open up our messages api this could become a problem
an alternative would be to implement robust message validation before or as part of processing them so that we're sure errors are not possible
far-in-the-future-messages:
we may want to just ignore these and leave them in the mempool until they're ready, so, the trigger filters out all messages from the future, potentially even deleting them from the mempool.
deletions (for future PR/issue)
probably need new kind of CRDT, similar to inserts
that crdt will have a message and processor that creates a seperate tombstone, in something like a tombstone table for each the record
that processor will also then deletes that record
update+insert crdt logic will need to be updated to check for tombstones, and discard messages for tombstoned records. For the update trigger, we'll also need a processor that discards messages for tombstoned records, since the main processor's trigger will not consume those messages due to it's check that filters out "missing records".
Metadata
Metadata
Assignees
Labels
Type
Projects
Status