Skip to content

prepare api for open access #89

@Geimaj

Description

@Geimaj

error handling todos:

invalid/failing message
depends on the reason for failing
if error message is due to invalidity (eg: upsert for a column that does not exist), we should delete it
consideration: we process messages in batched DB calls, so if there is just 1 invalid message in a batch, ideally the other messages should succeed and we should get a clear error pointing to the exact message(s) that failed so that we can then delete those messages. it's unclear if this exact functionality can be achieved through plain existing db queries (eg: some kind of conflict/error handling within the postgres query, and some kind of comprehensive, high fidelity error response). given this, we should investigate the scenario and improve our query to provide this, or if not possible, find an alternative.
for now, since all messages are in our control, we'll detect this quickly if a mistake is made and be able to remedy it before deploying to production, but as soon as we open up our messages api this could become a problem
an alternative would be to implement robust message validation before or as part of processing them so that we're sure errors are not possible
far-in-the-future-messages:

we may want to just ignore these and leave them in the mempool until they're ready, so, the trigger filters out all messages from the future, potentially even deleting them from the mempool.

deletions (for future PR/issue)

probably need new kind of CRDT, similar to inserts
that crdt will have a message and processor that creates a seperate tombstone, in something like a tombstone table for each the record
that processor will also then deletes that record
update+insert crdt logic will need to be updated to check for tombstones, and discard messages for tombstoned records. For the update trigger, we'll also need a processor that discards messages for tombstoned records, since the main processor's trigger will not consume those messages due to it's check that filters out "missing records".

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Next Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions