Leios design: Perf & Tracing #596
Conversation
docs/leios-design/README.md
Outdated
| As Leios development in all aspects is an ongoing and very much alive process, this chapter could only focus on | ||
| mid-to-high-level items. Depending on the progress of various protoypes and implementations, this high-level plan | ||
| will need to be broken down and sequenced into smaller tasks. We will do so keeping in mind the result of those | ||
| tasks should be reusable both internally, and by the community. |
There was a problem hiding this comment.
How should this relate to the implementation plan chapter (#592)?
There was a problem hiding this comment.
This section was an artifact of conflating the strategy description with design. Now that it's narrowed down what this PR targets, I've removed the section.
|
Generally, the issue this PR targets addresses "technical design and implementation plan" - so this contribution contains a bit of both. I'd be fine though with moving the contribution to issue #592. |
As discussed, let's incorporate the great points of this PR into respective sections next month - once the dust on the initial scaffold settles (the many in flight PRs onto #542). |
mgmeier
force-pushed
the
leios-design-perf-tracing
branch
from
69d8b0f to
05d3f14
Compare
mgmeier
force-pushed
the
leios-design-perf-tracing
branch
from
05d3f14 to
cabe84f
Compare
ch1bo
left a comment
ch1bo
left a comment
There was a problem hiding this comment.
Thanks for writing up these points again. They feel a bit disconnected from the existing sections and I'm not convinced that we should make a dedicated chapter about this (with the E2E testing for example still as part of technical design).
Would it make sense to incorporate these paragraphs with existing sections? See individual comments for some ideas.
docs/leios-design/README.md
Outdated
| ## Testing during development | ||
| Wheras simulations operate on models and are able to falsify hypotheses or assess probability of certain outcomes, evolving | ||
| prototypes and implementations rely on evidence to that end. A dedicated environment suitable for both performance and conformance testing will be created; primarily as feedbeck for development, but also to provide transparency into the ongoing process. |
There was a problem hiding this comment.
| prototypes and implementations rely on evidence to that end. A dedicated environment suitable for both performance and conformance testing will be created; primarily as feedbeck for development, but also to provide transparency into the ongoing process. | |
| prototypes and implementations rely on evidence to that end. A dedicated environment suitable for both performance and conformance testing will be created; primarily as feedback for development, but also to provide transparency into the ongoing process. |
docs/leios-design/README.md
Outdated
| isufficient to maintain that pressure. We will create a declarative, abstract definition of what constitues a workload to be submitted. This enables to pre-generate part or all submissions for the benchmarks. Moreover, it guarantees | ||
| identical outcomes regardless of how exactly the workload is generated. These workloads will retain their property of being customizable regarding particular aspects they stress in the system, such as the UTxO set, or Plutus script evaluation. | ||
| As raw data from a benchmark / confirmance test can be huge, existing analysis tooling will be extended or built, such that extracting key insights from raw data can be automated as much as possible. |
There was a problem hiding this comment.
| As raw data from a benchmark / confirmance test can be huge, existing analysis tooling will be extended or built, such that extracting key insights from raw data can be automated as much as possible. | |
| As raw data from a benchmark / conformance test can be huge, existing analysis tooling will be extended or built, such that extracting key insights from raw data can be automated as much as possible. |
docs/leios-design/README.md
Outdated
| this tends to be versatile and fast to evaluate incrementally. This means, system invariants can be tested as part of CI, or even by consuming live output of a running testnet. | ||
| Performance testing requires constant submission pressure over an extended period of time. With Leios being built for high throughput, creating submissions fully dynamically (as is the case with Praos benchmarks) is likely | ||
| isufficient to maintain that pressure. We will create a declarative, abstract definition of what constitues a workload to be submitted. This enables to pre-generate part or all submissions for the benchmarks. Moreover, it guarantees |
There was a problem hiding this comment.
| isufficient to maintain that pressure. We will create a declarative, abstract definition of what constitues a workload to be submitted. This enables to pre-generate part or all submissions for the benchmarks. Moreover, it guarantees | |
| insufficient to maintain that pressure. We will create a declarative, abstract definition of what constitutes a workload to be submitted. This enables to pre-generate part or all submissions for the benchmarks. Moreover, it guarantees |
| # Performance and quality assurance strategy | ||
| ## Observability as a first-class citizen |
There was a problem hiding this comment.
This could fit well as a section of the Overview (introduction) chapter?
| a specific prototype or implementation. This enables deploying adversarial nodes for the purpose of network conformance testing, as well as performance testing at system integration level. This environment also guarantees the | ||
| observed network behaviour or performance metrics have high confidence, and are reproducible. | ||
| Conformance testing can be done on multiple layers. For authoritative end-to-end verification of protocol states, all evidence will need to be processed wrt. the formal specification, keeping track of all states and transitions. A second, complementary |
There was a problem hiding this comment.
How does this paragraph relate to the section on "Correctness in two dimensions"? Should we incorporate the additional approach of using LTL there?
docs/leios-design/README.md
Outdated
| ## Testing during development | ||
| Wheras simulations operate on models and are able to falsify hypotheses or assess probability of certain outcomes, evolving | ||
| prototypes and implementations rely on evidence to that end. A dedicated environment suitable for both performance and conformance testing will be created; primarily as feedbeck for development, but also to provide transparency into the ongoing process. |
There was a problem hiding this comment.
Should this and the following paragraph maybe become part of the implementation plan, for example as part of Prototyping and adversarial testing?
| approach we chose is conformance testing using Linear Temporal Logic (LTL). By formulating LTL propositions that need to hold for observed evidence, one can achieve broad conformance and regression testing without embedding it in protocol semantics; | ||
| this tends to be versatile and fast to evaluate incrementally. This means, system invariants can be tested as part of CI, or even by consuming live output of a running testnet. | ||
| Performance testing requires constant submission pressure over an extended period of time. With Leios being built for high throughput, creating submissions fully dynamically (as is the case with Praos benchmarks) is likely |
There was a problem hiding this comment.
This sounds very related to what I already mentioned in Public testnets and integration on the implementation plan:
Performance testing measures achieved throughput against business requirements - sustained transaction rate, mempool-to-ledger latency, and behavior under bursty synthetic workloads.
Maybe we should elaborate there? Or Maybe even "before" on controlled environments / the Prototyping and adversarial testing
| This eventual step will stop support for prototypes and instead focus on full implementations of Leios. This will allow | ||
| for a uniform way to operate, and artificially constrain, Leios by configuration while maintaining its performance properties. | ||
| Furthermore, this phase will see custom benchmarks that can scale individual aspects of Leios independently (by config or protocol |
There was a problem hiding this comment.
The implementation plan is already outlining "phases" (which you hopefully agree with .. otherwise we should change them!) .. which of them would be about this kind of benchmarking? Is this "end-to-end benchmarking" corresponding to "end-to-end testing"?
| Note that the PoP checks probably are done at the certificate level, and that the above-described API should not be responsible for this. The current code on BLS12-381 already abstracts over both curves `G1`/`G2`, we should maintain this. The `BLST` package also exposes fast verification over many messages and signatures + public keys by doing a combined pairing check. This might be helpful, though it's currently unclear if we can use this speedup. It might be the case, since we have linear Leios, that this is never needed. | ||
| ## Performance & Tracing (P&T) |
There was a problem hiding this comment.
The content here is still very good and we should consider keeping it! Maybe we will (not need to be in this PR) expand it into doing what is mentioned there? e.g. drafting that specification of relevant traces and their semantics in this section of the "Technical design" chapter.
This PR contributes to #542 .This PR contributes to #615 .