[testnet] introduce new database schema and migration tooling.

[MathieuDutSik]

Motivation

The existing database schema is flawed because it consolidates all blobs, blob states, confirmed blocks, and events under a single root key (the shared one &[]).

We change this here.

Proposal

The new schema is chosen with the following goal in mind:

• The existing root_keys of ChainState and BlockExplorer remain in the same place.
• The partition keys are short: At most 33 bytes.
• The grouping is via CryptoHash, BlobId, ChainId, or index.
• The RootKey type is only used for getting the root_key in contrast to BaseKey, which serves two purposes: root_key for some variants, key for some others.

Of utmost importance is to have a viable migration path. This is done according to the following points:

• For the client, the migration is done automatically at startup. The same applies to indexers, exporters, and so on. This is possible because those applications do not share a storage.
• There is a special key indicating the working schema. For old schema, its absence indicates the state of the database.
• For servers and proxies, there is no way to migrate startup. So a migration tool linera-migration is introduced.
• At startup, for proxies and servers, we check whether the storage has been migrated. If not we stop right away.
• However, when running from a blank state, it works. We determine whether we are

For testing, a test test_storage_migration is introduced. It does the following:

• It creates a storage according to the old schema.
• It does the migration
• It reads the storage state and then checks for equality.
• This is done for RocksDB, ScyllaDB, and memory. This is not done for DynamoDB because of a problem with this database that will soon be addressed.

Of relevance is the annoying fact that root keys of the form &[] are not handled correctly for RocksDB and linera-storage-service. This turns out not to be a problem:

• Yes, doing a find_keys_by_prefix(&[]) on the shared partition would get you keys from other root keys. However, that use case does not occur.
• Instead, what we have is find_keys_by_prefix(&[x]) for x a byte that is NOT the first byte of other root keys. So it works.

Test Plan

The CI.

Release Plan

TestNet Conway is the target.

Links

None.

[afck]

Looks great! The only remaining potential blocker for me is the mem::size_of issue.

[bart-linera]

It would be good to point out that 0 and 6 correspond to ChainState and BlockExporter, respectively (if I understand what this is correctly... it's not obvious even with the comment, I'm afraid 😬 )

[afck]

Good catch! 👍