Deckard uses redis as a caching mechanism to store queues, priorities, etc. If the entire data is wiped, Deckard will reconstruct it and stay resillient.

Until then, even with thousands of requests per second and millions of messages, it was not necessary to use a Redis Cluster to scale. A single redis node was able to support very high workloads. This can be different for user's environments as they may need it.

A simple solution for different usages is to deploy different instances of deckard for each domain using a dedicated redis instance. This will be enough for the most use cases.

Since we use lua scripts to perform atomic operations in Redis we need to create integration tests using Redis Clusters to ensure that our lua scripts can work with this type of environment, as the documentation states:

Important: to ensure the correct execution of scripts, both in standalone and clustered deployments, all names of keys that a script accesses must be explicitly provided as input key arguments. The script should only access keys whose names are given as input arguments. Scripts should never access keys with programmatically-generated names or based on the contents of data structures stored in the database.

We may need to adapt or use hashing to be able to atomic update different keys to continue with the desired behavior we currently have for a single-node Redis instance.

I think we first need to study and plan how this would work, and any change made we must guarantee there is no breaking changes on the data structure, prioritizing the compatibility with previous versions of deckard.

This issue will first focus on creating a plan to improve this.