[marshal] Syncing blocks across epochs

[matthias-wright]

Hi all,
After the recent consensus update (congrats btw!), we are working on adding a dynamic validator set. Similar to the reshare example, we have an orchestrator that manages the Simplex instances for each epoch.
This works great when all validators start at the same time, but we would like to support the case where a validator starts later, when the other validators are already on another epoch.

I see a potential issue with this:

In marshal, if a finalize or notorize is received for an epoch that is not yet available in the scheme provider, it will respond false to the resolver, and the resolver will block the sender permanently.
One possible approach to support epochs is to optimistically accept messages from future epochs, and store them in a unverified store. When the new epoch becomes available in the scheme provider, the unverified blocks can be processed and verified.
The downside of this approach is that there is a ddos vector, because unverified blocks are written to disk.

Am I missing something here, and if not, are there plans to support this?

[patrick-ogrady]

After the recent consensus update (congrats btw!)

🙏 🙏 🙏 (all mechanisms will shortly support all schemes: #2220)

Similar to the reshare example, we have an orchestrator that manages the Simplex instances for each epoch. This works great when all validators start at the same time, but we would like to support the case where a validator starts later, when the other validators are already on another epoch.

Am I missing something here, and if not, are there plans to support this?

This is already supported in reshare via the Orchestrator channel:

monorepo/examples/reshare/src/orchestrator/actor.rs

Lines 200 to 242 in 8a1f604

	message = pending_backup.next() => {
	// If a message is received in an unregistered sub-channel in the pending network,
	// attempt to forward the orchestrator for the epoch.
	let Some((their_epoch, (from, _))) = message else {
	warn!("pending mux backup channel closed, shutting down orchestrator");
	break;
	};
	let Some(our_epoch) = engines.keys().last().copied() else {
	debug!(their_epoch, ?from, "received message from unregistered epoch with no known epochs");
	continue;
	};
	if their_epoch <= our_epoch {
	debug!(their_epoch, our_epoch, ?from, "received message from past epoch");
	continue;
	}
	// If we're not in the committee of the latest epoch we know about and we observe another
	// participant that is ahead of us, send a message on the orchestrator channel to prompt
	// them to send us the finalization of the epoch boundary block for our latest known epoch.
	if rate_limiter.check_key(&from).is_err() {
	continue;
	}
	let boundary_height = last_block_in_epoch(BLOCKS_PER_EPOCH, our_epoch);
	if self.marshal.get_finalization(boundary_height).await.is_some() {
	// Only request the orchestrator if we don't already have it.
	continue;
	};
	debug!(
	their_epoch,
	?from,
	"received backup message from future epoch, requesting orchestrator"
	);
	// Send the request to the orchestrator. This operation is best-effort.
	if orchestrator_sender.send(
	Recipients::One(from),
	UInt(our_epoch).encode().freeze(),
	true
	).await.is_err() {
	warn!("failed to send orchestrator request, shutting down orchestrator");
	break;
	}
	},

Basically, current validators observe a message from someone in an old epoch and send the boundary finalization to that player (which allows them to finish the epoch). Note: this only works when doing a full sync.

We plan to support something for state sync (where you just go to the nth epoch) but that'll likely just be providing the boundary finalization + list of validators (so should extend minimally).

One possible approach to support epochs is to optimistically accept messages from future epochs, and store them in a unverified store. When the new epoch becomes available in the scheme provider, the unverified blocks can be processed and verified.
The downside of this approach is that there is a ddos vector, because unverified blocks are written to disk.

We avoid this sort of "optimistic storage" approach for exactly the downside you mentioned. It becomes trivially to fill up your disk with junk.