Coinstack Fallback Process

Overview

In the case of an upstream provider outage, we want an easy way to fall back to our own self hosted infrastructure to prevent user downtime. Conversely, we also want an easy way to cut back over to the upstream provider once the outage has been resolved. To manage these flows, we are going to leverage CircleCI. This process assumes you have access to approve and/or rerun deployment workflows for Unchained.

General

• You can find the workflows under the unchained project on the main branch in CircleCI:
  

• The deployment pipeline will consist of two worflows:
  

  1. The standard deploy-main workflow to deploy new changes and optionally used to cut back over to the upstream provider in the case we had fallen back to our self hosted infrastructure.
  2. The new deploy-fallback workflow used to fall back to our self hosted infrastructure in the case of an upstream provider outage

• All workflow interaction will pertain to the most recent deployment which will be listed at the top by default

Fallback Process

• Select the deploy-fallback workflow:
  

• Approve the necessary coinstacks for fallback:
  

• Confirm the approval:
  

• Validate successful deployment to fallback infrastructure:
  

Fallback Process (Rerun)

• If the coinstack requiring fallback has already been run and the workflow is still in the "Needs Approval" stage, the workflow must be cancelled first before you are able to trigger a rerun:
  

• Rerun the deploy-fallback workflow:
  

• Continue following the Fallback Process above

Restore Process

• Rerun the deploy-main workflow:
  

• Select the deploy-main workflow now requiring approval:
  

• Approve all coinstacks - this will ensure everything is cut back over to the upstream provider and also ensure thedeploy-main workflow is left in a "Success" state:
  

• Validate successful deployment of the deploy-main workflow: