Synchronous history backfilling

[cjonas9]

What

This PR adds synchronous history backfilling of ledgers through the new configuration parameter backfill. This will fill the local SQL database with the most recent HISTORY_RETENTION_WINDOW ledgers, fetched from CDP. Usage: ./stellar-rpc --backfill, or one can specify this as a config parameter in the TOML or as an environmental variable.

Notes:

• After verifying the local DB state, this will backfill the specified number of ledgers from CDP backwards/"leftward", then backfill forwards/"rightward" up to the new most current ledger. By termination, we are guaranteed a database filled with the most recent HISTORY_RETENTION_WINDOW ledgers.
  • Leftward/backwards backfill: This backfills the range [oldest ledger we want in our DB <- oldest ledger in local DB-1] into the local DB. This phase is skipped if the DB is empty and this is generally slower than forwards backfill.
  • Rightward/forwards backfill: This backfills the range (max(newest ledger currently in local DB, oldest ledger we want in our DB) -> current tip of datastore] into the local DB. This phase is run twice if the DB is empty (on the second run, it just refreshes the current DB tip).
• After completion, it will start RPC to begin filling in the ledgers live through captive core. See my design document for details for how this is done.
• This design guarantees that, regardless of faults, the database will never be left with gaps in it. The client may supply a partially filled database, but backfill will error if this database initially has gaps in it. Note that (due to how the BufferedStorageBackend prevents backwards iteration) supplying a partially filled DB may be slower than the empty DB case if the partially filled database is 1.) minimally filled and/or 2.) close to the datastore current tip ledger (i.e. the larger the backwards backfilling run is, the slower this can be).
• The backfilled ledgers are ingested via CDP through the bufferedStorageBackend. It's datastore agnostic and has been tested on GCS and S3.
• Using the (rate-limited) public S3 datastore in the k8s soroban-rpc-pubnet-dev deployment, to ingest one week of ledgers (120,960 ledgers, ~150Gb), this takes about 3 hours.
  • Of three runs backfilling from no local DB, we see the following timing: 3h6m, 3h7m, 2h41m, 2h48m. I suspect that if any ledgers are in the DB (and a backwards phase runs), we would see longer runtimes.
  • In a truly pessimistic case (e.g. the backfill backwards phase is maximal), the runtime for this scenario is around ~3h45m.

Why

There is no way to backfill RPC's local database with historical ledgers, let alone a time-conscious method for this. This PR provides one.
See issues/discussions on this: #203, 1718

Known limitations

[N/A]

[Shaptic]

another pass

[Shaptic]

Amazing work!!