Merge pull request #641 from input-output-hk/ch1bo/demo-scaffold

Scaffold a demo/ directory

Author: ch1bo

demo/.envrc

@@ -0,0 +1 @@
+use nix

demo/2025-10/README.md

@@ -0,0 +1,227 @@
+# Prototype demo - October 2025
+
+Minimum viable demo of Leios network traffic interfering with Praos using a three node setup and prepared Praos and Leios data.
+
+> ![WARNING]
+> TODO: Add overview / architecture diagram.
+
+See https://github.com/IntersectMBO/ouroboros-consensus/issues/1701 for more context.
+
+## Prepare the shell environment
+
+- If your environment can successfully execute `cabal build exe:cardano-node` from this commit, then it can build this demo's exes.
+
+  ```
+  $ git log -1 10.5.1
+  commit ca1ec278070baf4481564a6ba7b4a5b9e3d9f366 (tag: 10.5.1, origin/release/10.5.1, nfrisby/leiosdemo2025-anchor)
+  Author: Jordan Millar <jordan.millar@iohk.io>
+  Date:   Wed Jul 2 08:24:11 2025 -0400
+
+      Bump node version to 10.5.1
+  ```
+
+- The Python script needs `pandas` and `matplotlib`.
+- The various commands and bash scripts below needs `toxiproxy`, `sqlite`, `ps` (which on a `nix-shell` might require the `procps` package for matching CLIB, eg), and so on.
+- Set `CONSENSUS_BUILD_DIR` to the absolute path of a directory in which `cabal build exe:immdb-server` will succeed.
+- Set `NODE_BUILD_DIR` to the absolute path of a directory in which `cabal build exe:cardano-node` will succeed.
+- Set `CONSENSUS_REPO_DIR` to the absolute path of the `ouroboros-consensus` repo.
+
+- Checkout a patched version of the `cardano-node` repository, something like the following, eg.
+
+```
+6119c5cff0 - (HEAD -> nfrisby/leiosdemo2025, origin/nfrisby/leiosdemo2025) WIP add Leios demo Consensus s-r-p (25 hours ago) <Nicolas Frisby>
+```
+
+- If you're using a `source-repository-package` stanza for the `cabal build exe:cardano-node` command in the `NODE_BUILD_DIR`, confirm that it identifies the `ouroboros-consensus` commit you want to use (eg the one you're reading this file in).
+
+## Build the exes
+
+```
+$ (cd $CONSENSUS_BUILD_DIR; cabal build exe:immdb-server exe:leiosdemo202510)
+$ IMMDB_SERVER="$(cd $CONSENSUS_BUILD_DIR; cabal list-bin exe:immdb-server)"
+$ DEMO_TOOL="$(cd $CONSENSUS_BUILD_DIR; cabal list-bin exe:leiosdemo202510)"
+$ (cd $CONSENSUS_BUILD_DIR; cabal build exe:cardano-node)
+$ CARDANO_NODE="$(cd $CONSENSUS_BUILD_DIR; cabal list-bin exe:cardano-node)"
+```
+
+## Prepare the input data files
+
+```
+$ (cd $CONSENSUS_BUILD_DIR; $DEMO_TOOL generate demoUpstream.db "${CONSENSUS_REPO_DIR}/demoManifest.json" demoBaseSchedule.json)
+$ cp demoBaseSchedule.json demoSchedule.json
+$ # You must now edit demoSchedule.json so that the first number in each array is 182.9
+$ echo '[]' >emptySchedule.json
+$ # create the following symlinks
+$ (cd $CONSENSUS_REPO_DIR; ls -l $(find nix/ -name genesis-*.json))
+lrwxrwxrwx 1 nfrisby nifr 30 Oct 24 16:27 nix/leios-mvd/immdb-node/genesis-alonzo.json -> ../genesis/genesis.alonzo.json
+lrwxrwxrwx 1 nfrisby nifr 29 Oct 24 16:27 nix/leios-mvd/immdb-node/genesis-byron.json -> ../genesis/genesis.byron.json
+lrwxrwxrwx 1 nfrisby nifr 30 Oct 24 16:27 nix/leios-mvd/immdb-node/genesis-conway.json -> ../genesis/genesis.conway.json
+lrwxrwxrwx 1 nfrisby nifr 31 Oct 24 16:27 nix/leios-mvd/immdb-node/genesis-shelley.json -> ../genesis/genesis.shelley.json
+lrwxrwxrwx 1 nfrisby nifr 30 Oct 24 16:27 nix/leios-mvd/leios-node/genesis-alonzo.json -> ../genesis/genesis.alonzo.json
+lrwxrwxrwx 1 nfrisby nifr 29 Oct 24 16:27 nix/leios-mvd/leios-node/genesis-byron.json -> ../genesis/genesis.byron.json
+lrwxrwxrwx 1 nfrisby nifr 30 Oct 24 16:27 nix/leios-mvd/leios-node/genesis-conway.json -> ../genesis/genesis.conway.json
+lrwxrwxrwx 1 nfrisby nifr 31 Oct 24 16:27 nix/leios-mvd/leios-node/genesis-shelley.json -> ../genesis/genesis.shelley.json
+```
+
+## Prepare to run scenarios
+
+Ensure a toxiproxy server is running.
+
+```
+$ toxiproxy-server 1>toxiproxy.log 2>&1 &
+```
+
+## Run the scenario
+
+Run the scenario with `emptySchedule.json`, ie no Leios traffic.
+
+```
+$ LEIOS_UPSTREAM_DB_PATH="$(pwd)/demoUpstream.db" LEIOS_SCHEDULE="$(pwd)/emptySchedule.json" SECONDS_UNTIL_REF_SLOT=5 REF_SLOT=177 CLUSTER_RUN_DATA="${CONSENSUS_REPO_DIR}/nix/leios-mvd" CARDANO_NODE=$CARDANO_NODE IMMDB_SERVER=$IMMDB_SERVER ${CONSENSUS_REPO_DIR}/scripts/leios-demo/leios-october-demo.sh
+$ # wait about ~20 seconds before stopping the execution by pressing any key
+```
+
+Run the scenario with `demoSchedule.json`.
+
+```
+$ LEIOS_UPSTREAM_DB_PATH="$(pwd)/demoUpstream.db" LEIOS_SCHEDULE="$(pwd)/demoSchedule.json" SECONDS_UNTIL_REF_SLOT=5 REF_SLOT=177 CLUSTER_RUN_DATA="${CONSENSUS_REPO_DIR}/nix/leios-mvd" CARDANO_NODE=$CARDANO_NODE IMMDB_SERVER=$IMMDB_SERVER ${CONSENSUS_REPO_DIR}/scripts/leios-demo/leios-october-demo.sh
+$ # wait about ~20 seconds before stopping the execution by pressing any key
+```
+
+## Analysis
+
+Compare and contrast the `latency_ms` column for the rows with a slot that's after the reference slot 177.
+The first few such ros (ie those within a couple seconds of the reference slot) seem to often also be disrupted, because the initial bulk syncing to catch up to the reference slot presumably leaves the node in a disrupted state for a short interval.
+
+**WARNING**.
+Each execution consumes about 0.5 gigabytes of disk.
+The script announces where (eg `Temporary data stored at: /run/user/1000/leios-october-demo.c5Wmxc`), so you can delete each run's data when necessary.
+
+**INFO**.
+If you don't see any data in the 'Extracted and Merged Data Summary' table, then check the log files in the run's temporary directory.
+This is where you might see messages about, eg, the missing `genesis-*.json` files, bad syntax in the `demoSchedule.json` file, etc.
+
+# Details about the demo components
+
+## The topology
+
+For this first iteration, the demo topology is a simple linear graph.
+
+```mermaid
+flowchart TD
+    MockedUpstreamPeer --> Node0 --> MockedDownstreamPeer
+```
+
+**INFO**.
+In this iteration of the demo, the mocked downstream peer (see section below) is simply another node, ie Node1.
+
+## The Praos traffic and Leios traffic
+
+In this iteration of the demo, the data and traffic is very simple.
+
+- The Praos data is a simple chain provided by the Performance&Tracing team.
+- The mocked upstream peer serves each Praos block when the mocked wall-clock reaches the onset of their slots.
+- The Leios data is ten 12.5 megabyte EBs.
+  They use the minimal number of txs necessary in order to accumulate 12.5 megabytes in order to minimize the CPU&heap overhead of the patched-in Leios logic, since this iteration of trhe demo is primarily intended to focus on networking.
+- The mocked upstream peer serves those EBs just prior to the onset of one of the Praos block's slot, akin to (relatively minor) ATK-LeiosProtocolBurst attack.
+  Thus, the patched nodes are under significant Leios load when that Praos block begins diffusing.
+
+## The demo tool
+
+The `cabal run exe:leiosdemo202510 -- generate ...` command generates a SQLite database with the following schema.
+
+```
+CREATE TABLE ebPoints (
+    ebSlot INTEGER NOT NULL
+  ,
+    ebHashBytes BLOB NOT NULL
+  ,
+    ebId INTEGER NOT NULL
+  ,
+    PRIMARY KEY (ebSlot, ebHashBytes)
+  ) WITHOUT ROWID;
+CREATE TABLE ebTxs (
+    ebId INTEGER NOT NULL   -- foreign key ebPoints.ebId
+  ,
+    txOffset INTEGER NOT NULL
+  ,
+    txHashBytes BLOB NOT NULL   -- raw bytes
+  ,
+    txBytesSize INTEGER NOT NULL
+  ,
+    txBytes BLOB   -- valid CBOR
+  ,
+    PRIMARY KEY (ebId, txOffset)
+  ) WITHOUT ROWID;
+```
+
+The contents of the generated database are determine by the given `manifest.json` file.
+For now, see the `demoManifest.json` file for the primary schema: each "`txRecipe`" is simply the byte size of the transaction.
+
+The `generate` subcommand also generates a default `schedule.json`.
+Each EB will have two array elements in the schedule.
+The first number in an array element is a fractional slot, which determines when the mocked upstream peer will offer the payload.
+The rest of the array element is `MsgLeiosBlockOffer` if the EB's byte size is listed or `MsgLeiosBlockTxsOffer` if `null` is listed.
+
+The secondary schema of the manifest allows for EBs to overlap (which isn't necessary for this demo, despite the pathced node fully supporting it).
+Overlap is created by an alternative "`txRecipe`", an object `{"share": "XYZ", "startIncl": 90, "stopExcl": 105}` where `"nickname": "XYZ"` was included in a preceding _source_ EB recipe.
+The `"startIncl`" and `"stopExcl"` are inclusive and exclusive indices into the source EB (aka a left-closed right-open interval); `"stopExcl"` is optional and defaults to the length of the source EB.
+With this `"share"` syntax, it is possible for an EB to include the same tx multiple times.
+That would not be a well-formed EB, but the prototype's behavior in response to such an EB is undefined---it's fine for the prototype to simply assume all the Leios EBs and txs in their closures are well-formed.
+(TODO check for this one, since it's easy to check for---just in the patched node itself, or also in `generate`?)
+
+## The mocked upstream peer
+
+The mocked upstream peer is a patched variant of `immdb-server`.
+
+- It runs incomplete variants of LeiosNotify and LeiosFetch: just EBs and EB closures, nothing else (no EB announcements, no votes, no range requests).
+- It serves the EBs present in the given `--leios-db`; it sends Leios notificaitons offering the data according to the given `--leios-schedule`.
+  See the demo tool section above for how to generate those files.
+
+## The patched node/node-under-test
+
+The patched node is a patched variant of `cardano-node`.
+All of the material changes were made in the `ouroboros-consensus` repo; the `cardano-node` changes are merely for integration.
+
+- It runs the same incomplete variants of LeiosNotify and LeiosFetch as the mocked upstream peer.
+- The Leios fetch request logic is a fully fledged first draft, with following primary shortcomings.
+  - It only handles EBs and EB closures, not votes and not range requests.
+  - It retains a number of heap objects in proportion with the number of txs in EBs it has acquired.
+    The real node---and so subsequent iterations of this prototype---must instead keep that data on disk.
+    This first draft was intended to do so, but we struggled to invent the fetch logic algorithm with the constraint that some of its state was on-disk; that's currently presumed to be possible, but has been deferred to a subsequent iteration of the prototype.
+  - It never discards any information.
+    The real node---and so subsequent iterations of this prototype---must instead discard EBs and EB closures once they're old enough, unless they are needed for the immutable chain.
+  - Once it decides to fetch a set of txs from an upstream peer for the sake of some EB closure(s), it does not necessarily compose those into an optimal set of requests for that peer.
+    We had not identified the potential for an optimizing algorithm here until writing this first prototype, so it just does something straight-forward and naive for now (which might be sufficient even for the real-node---we'll have to investigate later).
+
+There are no other changes.
+In particular, that means the `ouroboros-network` mux doesn't not deprioritize Leios traffic.
+That change is an example of what this first prototype is intended to potentially demonstrate the need for.
+There are many such changes, from small to large.
+Some examples includes the following.
+
+- The prototype uses SQLite3 with entirely default settings.
+  Maybe Write-Ahead Log mode would be much preferable, likely need to VACUUM at some point, and so on.
+- The prototype uses a mutex to completely isolate every SQLite3 invocation---that's probably excessive, but was useful for some debugging during initial development (see the Engineering Notes appendix)
+- The prototype chooses several _magic numbers_ for resource utilization limits (eg max bytes per reqeust, max outsanding bytes per peer, fetch decision logic rate-limiting, txCache disk-bandwidth rate-limiting, etc).
+  These all ultimately need to be tuned for the intended behvaiors on `mainnet`.
+- The prototype does not deduplicate the storage of EBs' closures when they share txs.
+  This decision makes the LeiosFetch server a trivial single-pass instead of a join.
+  However, it "wastes" disk space and disk bandwidth.
+  It's left to future work to decide whether that's a worthwhile trade-off.
+
+## The mocked downstream node
+
+For simplicity, this is simply another instance of the patched node.
+In the future, it could be comparatively lightweight and moreover could replay an arbitrary schedule of downstream requests, dual to the mocked upstream peer's arbitrary schedule of upstream notifications.
+
+# Appendix: Engineering Notes
+
+This section summarizes some lessons learned during the development of this prototype.
+
+- Hypothesis: A SQLite connection will continue to hold SQLite's internal EXCLUSIVE lock _even after the transaction is COMMITed_ when the write transaction involved a prepared statement that was accidentally not finalized.
+  That hypothesis was inferred from a painstaking debugging session, but I haven't not yet confirmed it in isolation.
+  The bugfix unsuprisingly amounted to using `bracket` for all prepare/finalize pairs and all BEGIN/COMMIT pairs; thankfully our DB patterns seem to accommodate such bracketing.
+- The SQLite query plan optimizer might need more information in order to be reliable.
+  Therefore at least one join (the one that copies out of `txCache` for the EbTxs identified in an in-memory table) was replaced with application-level iteration.
+  It's not yet clear whether a one-time ANALYZE call might suffice, for example.
+  Even if it did, it's also not yet clear how much bandwidth usage/latency/jitter/etc might be reduced.

demo/2025-11/README.md

@@ -0,0 +1,81 @@
+# Prototype demo - November 2025
+
+Slight improvement of the [October 2025 demonstration](../2025-10) using `tc` and better observability using Grafana et al. In summary the progress made during November 2025 on the Leios networking prototype is:
+
+- The surprisingly-high latency observed in the October demo was explained and reined in.
+- Key structured log events were added to the prototype.
+- Observability/reporting/monitoring was improved.
+- Packaging of the prerequisites for executing the demo was improved.
+
+![](./demo-2025-11.excalidraw.svg)
+
+> [!TIP]
+> This is an excalidraw SVG with embedded scene so it can be loaded and edited in [https://excalidraw.com/].
+
+## Bufferbloat
+
+The investigation into the unexpectedly high latency seen in October and related refinements to the prototype are apparent in the asynchronous conversation that took place in the comments on this tracking Issue https://github.com/IntersectMBO/ouroboros-consensus/issues/1756.
+
+- The latency was due to https://www.bufferbloat.net.
+  In October, the bufferbloat arose directly from the naive use of [Toxiproxy](https://github.com/Shopify/toxiproxy) for the initial demo.
+- As user-space mechanism, Toxiproxy cannot introduce latency/rate/etc in a way that will influence the kernel algorithms managing the TCP stream.
+- [Linux Traffic Control](https://tldp.org/HOWTO/Traffic-Control-HOWTO/intro.html) is the approriate mechanism.
+- An example of relevant commands for a more appropriate WAN (Wide Area Network) emulation can be found in this GitHub comment https://github.com/IntersectMBO/ouroboros-consensus/issues/1756#issuecomment-3587268042.
+  - `htb rate 100mbt` limts the sender's bandwidth.
+  - `fq_codel` paces the sender's traffic, adapting to any bottleneck between it and the recipient.
+  - `netem delay` established the link latency of 20ms between `fq_codel` and the recipient.
+- The Networking Team is now taking over this aspect of the setup for future Leios demos, refining//enriching the WAN emulation, preparing some testing on actual WANs (physically separated machines), and considering which mechniams ought to mitigate the risk of Leios-induced bufferbloat (perhaps as part of an ATK-LeiosProtocolProtocolBurst) increasing the latency of Praos traffic once Leios is deployed on mainnet.
+
+## Improved Logging
+
+Additional key events in both the mocked upstream peer and the node-under-test are now emitted as structured JSON, which the demo's analysis script processes.
+Highlights include the following.
+
+- The node reliably indicates when concludes it acquired the last of the txs it was missing from an EB.
+  In particular, this event is raised then a MsgLeiosBlockTx arrives with all the txs that the node was missing from some EB.
+  Even if the final tx were to arrive as part of a separate EB, this event would still be emitted for each EB that the MsgLeiosBlockTx completes.
+- Both the node and upstream peer report when ChainSync's MsgRollForward, BlockFetch's MsgRequestRange and MsgBlock, and Leios's MsgLeiosBlockRequest, MsgLeiosBlock, MsgLeiosBlockTxsRequest, and MsgLeiosBlockTxs are sent and received.
+  The demo's analysis script displays a table of when these messages were sent and received.
+  This table very usefully indicates how much to alter the timings in the `demoSchedule.json` file in order to change which parts of the Praos traffic a particular EB's exchange overlaps with.
+- A patch to `ouroboros-network` was introduced to allow two additional timings, which will are expected to help make subsequent visualizations of the message exchange more accurate.
+  - When a mini protocol begins trying to enqueue a message in the mux, even if the mux is unable to accept that message immediately.
+  - When the demux receives the last byte of some message, even if the mini protocol doesn't attempt to decode that message immediately.
+- The `ss` tool is being used to sample socket statistics throughout the demo's execution, so that the TCP algorithm's state can be monitored.
+  For example, the `rtt` and `notsent` fields are directly related to bufferbloat.
+
+## Monitoring with Grafana
+
+TODO
+
+## Packaging
+
+TODO
+
+## Building from Source
+
+Contributers who want to build the demo from source will need to packages in the three repositories on these commits.
+No other packages have yet been patched for this demo, the appropriate versions are those used in the 10.5.1 build.
+Beware that the listed commits do not already include `source-repository-package` stanzas in their `cabal.project` files, if that's
+the contributor's chosen method for cross-repo dependencies.
+
+```
+$ for i in ouroboros-consensus ouroboros-network cardano-node; do (cd $i; echo REPO $i; git log -1); done
+REPO ouroboros-consensus
+commit 7929c3716a18abb852f8abec7111c78f2059287e (HEAD -> nfrisby/leios-202511-demo, origin/nfrisby/leios-202511-demo)
+Author: Nicolas Frisby <nick.frisby@iohk.io>
+Date:   Thu Nov 27 12:57:43 2025 -0800
+
+    leiosdemo202511: polishing per-message table format
+REPO ouroboros-network
+commit 479f0d0d82413162c8444b912394dd74c052831f (HEAD -> nfrisby/leios-202511-demo, tag: leios-202511-demo, origin/nfrisby/leios-202511-demo)
+Author: Nicolas Frisby <nick.frisby@iohk.io>
+Date:   Thu Nov 27 10:49:49 2025 -0800
+
+    leiosdemo202511: introduce BearerBytes class
+REPO cardano-node
+commit 93d2c8481912309faf5a7d9058f9fdeca95710a0 (HEAD -> nfrisby/leios-202511-demo, origin/nfrisby/leios-202511-demo)
+Author: Nicolas Frisby <nick.frisby@iohk.io>
+Date:   Thu Nov 27 11:02:11 2025 -0800
+
+    leiosdemo202511: integrate ouroboros-network BearerBytes
+```

demo/2025-11/demo-2025-11.excalidraw.svg

@@ -0,0 +1,12 @@
+# Demos
+
+This is a collection of Leios demonstrations created using specially patched versions of `cardano-node` and other components not originating from this repository:
+
+- [2025-10](2025-10/): Minimum viable demo of Leios network traffic interfering with Praos
+- [2025-11](2025-11/): Improvement of MVD using tc and better observability
+
+There are other, component-specific demos you might be looking for:
+
+- [Visualizer](../ui) contains a few stored scenarios that are standalone demos
+- [Cryptography](../crypto-benchmarks.rs/demo) demo of signing/verifying votes and certificates
+- [Trace translator](../scripts/trace-translator/demo) has an example/demo

demo/README.md

@@ -1 +1 @@
-use nix
+use nix .#web