diff --git a/docs/filling_tests/test_ids.md b/docs/filling_tests/test_ids.md index 7005ab639b..069518bb06 100644 --- a/docs/filling_tests/test_ids.md +++ b/docs/filling_tests/test_ids.md @@ -55,8 +55,8 @@ The test framework can also generate blockchain tests containing blocks that spa Each Python test case is also typically parametrized by test type, respectively fixture format. For example, if the test is implemented as a `state_test`, the test framework will additionally generate the following blockchain test fixtures (consisting of a single block with a single transaction): -- a `blockchain_test` which can be tested via the Hive `eest/consume-rlp` simulator (or directly via a dedicated client interface). -- a `blockchain_engine_test` (for post-merge forks) which can be tested via the Hive `eest/consume-engine` simulator. +- a `blockchain_test` which can be tested via the Hive `eels/consume-rlp` simulator (or directly via a dedicated client interface). +- a `blockchain_engine_test` (for post-merge forks) which can be tested via the Hive `eels/consume-engine` simulator. ### Example: The Test IDs generated for `test_chainid` diff --git a/docs/running_tests/consume/exceptions.md b/docs/running_tests/consume/exceptions.md index 453284aedf..a5a82c499a 100644 --- a/docs/running_tests/consume/exceptions.md +++ b/docs/running_tests/consume/exceptions.md @@ -98,7 +98,7 @@ uv run consume engine --disable-strict-exception-matching=nimbus-el Enable verbose client output: ```bash -./hive --sim ethereum/eest/consume-engine \ +./hive --sim ethereum/eels/consume-engine \ --docker.output \ --sim.loglevel 5 ``` diff --git a/docs/running_tests/execute/hive.md b/docs/running_tests/execute/hive.md index 128938a29c..790516ecff 100644 --- a/docs/running_tests/execute/hive.md +++ b/docs/running_tests/execute/hive.md @@ -2,12 +2,12 @@ Tests can be executed on a local hive-controlled single-client network by running the `execute hive` command. -## The `eest/execute-blobs` Simulator +## The `eels/execute-blobs` Simulator The `blob_transaction_test` execute test spec sends blob transactions to a running client in order to verify its `engine_getBlobsVX` endpoint behavior. These tests can be run using: ```bash -./hive --client besu --client-file ./configs/osaka.yaml --sim ethereum/eest/execute-blobs +./hive --client besu --client-file ./configs/osaka.yaml --sim ethereum/eels/execute-blobs ``` See [Hive](../hive/index.md) for help installing and configuring Hive. diff --git a/docs/running_tests/hive/client_config.md b/docs/running_tests/hive/client_config.md index be04107f11..e2713780e9 100644 --- a/docs/running_tests/hive/client_config.md +++ b/docs/running_tests/hive/client_config.md @@ -102,17 +102,17 @@ cp -r /path/to/your/go-ethereum ./clients/go-ethereum/go-ethereum-local Force rebuild base images: ```bash -./hive --docker.pull --sim ethereum/eest/consume-engine +./hive --docker.pull --sim ethereum/eels/consume-engine ``` Force rebuild specific client: ```bash -./hive --docker.nocache "clients/go-ethereum" --sim ethereum/eest/consume-engine +./hive --docker.nocache "clients/go-ethereum" --sim ethereum/eels/consume-engine ``` Show the docker container build output: ```bash -./hive --docker.buildoutput --sim ethereum/eest/consume-engine +./hive --docker.buildoutput --sim ethereum/eels/consume-engine ``` diff --git a/docs/running_tests/hive/common_options.md b/docs/running_tests/hive/common_options.md index ff3d313252..2272e7d058 100644 --- a/docs/running_tests/hive/common_options.md +++ b/docs/running_tests/hive/common_options.md @@ -1,15 +1,15 @@ # Common Simulator Options -All EEST Hive simulators share common command-line options and patterns. +All execution-specs (EELS) Hive simulators share common command-line options and patterns. ## Basic Usage -While they may be omitted, it's recommended to specify the `fixtures` and `branch` simulator build arguments when running EEST simulators. +While they may be omitted, it's recommended to specify the `fixtures` and `branch` simulator build arguments when running execution-specs simulators. For example, this runs "stable" fixtures from the v4.3.0 [latest stable release](../releases.md#standard-releases) and builds the simulator at the v4.3.0 tag: ```bash -./hive --sim ethereum/eest/consume-engine \ +./hive --sim ethereum/eels/consume-engine \ --sim.buildarg fixtures=stable@v4.3.0 \ --sim.buildarg branch=v4.3.0 \ --client go-ethereum @@ -20,7 +20,7 @@ For example, this runs "stable" fixtures from the v4.3.0 [latest stable release] Run a subset of tests by filtering tests using `--sim.limit=` to perform a regular expression match against test IDs: ```bash -./hive --sim ethereum/eest/consume-engine --sim.limit ".*eip4844.*" +./hive --sim ethereum/eels/consume-engine --sim.limit ".*eip4844.*" ``` ### Collect Only/Dry-Run @@ -28,7 +28,7 @@ Run a subset of tests by filtering tests using `--sim.limit=` to perform The `collectonly:` prefix can be used to inspect which tests would match an expression (dry-run), `--docker.output` must be specified to see the simulator's collection result: ```bash -./hive --sim ethereum/eest/consume-engine \ +./hive --sim ethereum/eels/consume-engine \ --sim.buildarg fixtures=stable@v4.3.0 \ --sim.buildarg branch=v4.3.0 \ --docker.output \ @@ -40,7 +40,7 @@ The `collectonly:` prefix can be used to inspect which tests would match an expr The `id:` prefix can be used to select a single test via its ID (this will automatically escape any special characters in the test case ID): ```console -./hive --sim ethereum/eest/consume-engine \ +./hive --sim ethereum/eels/consume-engine \ --sim.buildarg fixtures=stable@v4.3.0 \ --sim.buildarg branch=v4.3.0 \ --docker.output \ @@ -52,7 +52,7 @@ The `id:` prefix can be used to select a single test via its ID (this will autom To run multiple tests in parallel, use `--sim.parallelism`: ```bash -./hive --sim ethereum/eest/consume-rlp --sim.parallelism 4 +./hive --sim ethereum/eels/consume-rlp --sim.parallelism 4 ``` ### Output Options @@ -60,7 +60,7 @@ To run multiple tests in parallel, use `--sim.parallelism`: See hive log output in the console: ```bash -./hive --sim ethereum/eest/consume-engine --sim.loglevel 5 +./hive --sim ethereum/eels/consume-engine --sim.loglevel 5 ``` ### Container Issues @@ -68,5 +68,5 @@ See hive log output in the console: Increase client timeout: ```bash -./hive --client.checktimelimit=180s --sim ethereum/eest/consume-engine +./hive --client.checktimelimit=180s --sim ethereum/eels/consume-engine ``` diff --git a/docs/running_tests/hive/dev_mode.md b/docs/running_tests/hive/dev_mode.md index 61b382ff44..965a7e1855 100644 --- a/docs/running_tests/hive/dev_mode.md +++ b/docs/running_tests/hive/dev_mode.md @@ -1,6 +1,6 @@ # Hive Development Mode -This section explains how to run EEST simulators using their EEST commands, e.g., `uv run consume engine`, against a Hive "development" server as apposed to using the standalone `./hive` command. +This section explains how to run EELS simulators using their Python-based commands, e.g., `uv run consume engine`, against a Hive "development" server as apposed to using the standalone `./hive` command. This avoids running the simulator in a dockerized environment and has several advantages: @@ -18,7 +18,7 @@ This avoids running the simulator in a dockerized environment and has several ad ### Prerequisites -- EEST is installed, see [Installation](../../getting_started/installation.md) +- The execution-specs repo is setup in development mode, see [Installation](../../getting_started/installation.md) - Hive is built, see [Hive](../hive/index.md#quick-start). ## Hive Dev Setup on Linux @@ -29,7 +29,7 @@ This avoids running the simulator in a dockerized environment and has several ad ./hive --dev --client go-ethereum --client-file clients.yaml --docker.output ``` -2. In a separate shell, configure environment for EEST: +2. In a separate shell, configure environment for execution-specs: === "bash/zsh" @@ -43,7 +43,7 @@ This avoids running the simulator in a dockerized environment and has several ad set -x HIVE_SIMULATOR http://127.0.0.1:3000 ``` -3. Run EEST consume commands +3. Run execution-specs `consume` commands ```bash uv run consume engine --input ./fixtures -k "test_chainid" @@ -56,37 +56,36 @@ Due to Docker running within a VM on macOS, the host machine and Docker containe 1. Linux VM: Run a Linux virtual machine on your macOS and execute the standard development workflow above from within the VM. 2. Remote Linux: SSH into a remote Linux environment (server or cloud instance) and run development mode there. -3. **Docker Development Image**: Create a containerized EEST environment that runs within Docker's network namespace (recommended). +3. **Docker Development Image**: Create a containerized execution-specs environment that runs within Docker's network namespace (recommended). The following section details the setup and usage of option 3. -### EEST Docker Development Image +### EELS Docker Development Image -Within the [`eest/`](https://github.com/ethereum/hive/tree/master/simulators/ethereum/eest) directory of hive, a new dockerfile must be created: `Dockerfile.dev`, with the following contents: +Within the [`eels/`](https://github.com/ethereum/hive/tree/master/simulators/ethereum/eels) directory of hive, a new dockerfile must be created: `Dockerfile.dev`, with the following contents: ```docker FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim -ARG branch=main -ENV GIT_REF=${branch} +ARG branch="" RUN apt-get update && apt-get install -y git -RUN git init execution-spec-tests && \ - cd execution-spec-tests && \ - git remote add origin https://github.com/ethereum/execution-spec-tests.git && \ - git fetch --depth 1 origin $GIT_REF && \ - git checkout FETCH_HEAD; - -WORKDIR /execution-spec-tests +RUN git clone --depth 1 https://github.com/ethereum/execution-specs.git && \ + cd execution-specs && \ + if [ -n "$branch" ]; then \ + git fetch --depth 1 origin "$branch" && \ + git checkout FETCH_HEAD; \ + fi +WORKDIR /execution-specs/packages/testing RUN uv sync ENTRYPOINT ["/bin/bash"] ``` -This dockerfile will be our entry point for running EEST commands. +This dockerfile will be our entry point for running simulator commands. -### `eest/` Hive Directory Structure +### `eels/` Hive Directory Structure ```tree -├── eest +├── eels │ ├── Dockerfile.dev │ ├── consume-block-rlp │ │ └── Dockerfile @@ -108,10 +107,10 @@ This dockerfile will be our entry point for running EEST commands. ./hive --dev --dev.addr :3000 --client go-ethereum --client-file clients.yaml ``` -3. In a separate terminal session, build the EEST development image: +3. In a separate terminal session, build the EELS development image: ```bash - cd simulators/ethereum/eest/ + cd simulators/ethereum/eels/ docker build -t macos-consume-dev -f Dockerfile.dev . ``` @@ -136,7 +135,7 @@ When Hive runs in dev mode: 3. Keeps the Hive Proxy container running between test executions. 4. Waits for external simulator connections via the API. -This allows EEST's consume commands to connect to the running Hive instance and execute tests interactively. +This allows the EELS's consume commands to connect to the running Hive instance and execute tests interactively. ## More Options Available diff --git a/docs/running_tests/hive/index.md b/docs/running_tests/hive/index.md index 81f75be5ee..aba902eb49 100644 --- a/docs/running_tests/hive/index.md +++ b/docs/running_tests/hive/index.md @@ -1,6 +1,6 @@ # Hive -@ethereum/hive is a containerized testing framework that helps orchestrate test execution against Ethereum clients. Hive is incredibly extensible; new test suites can be implemented in a module manner as "simulators" that interact with clients to test certain aspects of their behavior. EEST implements several simulators, see [Running Tests](../running.md) for an overview. +@ethereum/hive is a containerized testing framework that helps orchestrate test execution against Ethereum clients. Hive is incredibly extensible; new test suites can be implemented in a module manner as "simulators" that interact with clients to test certain aspects of their behavior. The execution-specs `testing` package implements several simulators, see [Running Tests](../running.md) for an overview. ## Quick Start diff --git a/docs/running_tests/releases.md b/docs/running_tests/releases.md index 2136f0401a..295fd9c64e 100644 --- a/docs/running_tests/releases.md +++ b/docs/running_tests/releases.md @@ -7,10 +7,10 @@ | Format | Consumed by the client | Location in `.tar.gz` release | | -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | | [State Tests](./test_formats/state_test.md) | - directly via a `statetest`-like command
(e.g., [go-ethereum/cmd/evm/staterunner.go](https://github.com/ethereum/go-ethereum/blob/4bb097b7ffc32256791e55ff16ca50ef83c4609b/cmd/evm/staterunner.go)) | `./fixtures/state_tests/` | -| [Blockchain Tests](./test_formats/blockchain_test.md) | - directly via a `blocktest`-like command
(e.g., [go-ethereum/cmd/evm/blockrunner.go](https://github.com/ethereum/go-ethereum/blob/4bb097b7ffc32256791e55ff16ca50ef83c4609b/cmd/evm/blockrunner.go))
- using the [RLPeest/consume-rlp Simulator](./running.md#rlp) via block import | `./fixtures/blockchain_tests/` | -| [Blockchain Engine Tests](./test_formats/blockchain_test_engine.md) | - using the [eest/consume-engine Simulator](./running.md#engine) and the Engine API | `./fixtures/blockchain_tests_engine/` | +| [Blockchain Tests](./test_formats/blockchain_test.md) | - directly via a `blocktest`-like command
(e.g., [go-ethereum/cmd/evm/blockrunner.go](https://github.com/ethereum/go-ethereum/blob/4bb097b7ffc32256791e55ff16ca50ef83c4609b/cmd/evm/blockrunner.go))
- using the [eels/consume-rlp Simulator](./running.md#rlp) via block import | `./fixtures/blockchain_tests/` | +| [Blockchain Engine Tests](./test_formats/blockchain_test_engine.md) | - using the [eels/consume-engine Simulator](./running.md#engine) and the Engine API | `./fixtures/blockchain_tests_engine/` | | [Transaction Tests](./test_formats/transaction_test.md) | - using a new simulator coming soon | None; executed directly from Python source,
using a release tag | -| Blob Transaction Tests | - using the [eest/execute-blobs Simulator](./execute/hive.md#the-eestexecute-blobs-simulator) and | None; executed directly from Python source,
using a release tag | +| Blob Transaction Tests | - using the [eels/execute-blobs Simulator](./execute/hive.md#the-eestexecute-blobs-simulator) and | None; executed directly from Python source,
using a release tag | ## Release URLs and Tarballs diff --git a/docs/running_tests/running.md b/docs/running_tests/running.md index 94272d7d4e..efd98c340d 100644 --- a/docs/running_tests/running.md +++ b/docs/running_tests/running.md @@ -23,9 +23,9 @@ Both `consume` and `execute` provide sub-commands which correspond to different The following sections describe the different methods in more detail. -!!! note "`./hive --sim=eest/consume-engine` vs `consume engine`" +!!! note "`./hive --sim=eels/consume-engine` vs `consume engine`" - EEST simulators can be ran either standalone using the `./hive` command or via an EEST command against a `./hive --dev` backend, more details are [provided below](#two-methods-to-run-eest-simulators). + The execution-specs simulators can be ran either standalone using the `./hive` command or via a `uv`/Python-based command against a `./hive --dev` backend, more details are [provided below](#two-methods-to-run-eels-simulators). ## Direct @@ -48,7 +48,7 @@ The EEST `consume direct` command is a small wrapper around client direct interf | Nomenclature | | | -------------- | ------------------------ | | Command | `consume engine` | -| Simulator | `eest/consume-engine` | +| Simulator | `eels/consume-engine` | | Fixture format | `blockchain_test_engine` | The consume engine method tests execution clients via the Engine API by sending block payloads and verifying the response (post-merge forks only). This method provides the most realistic testing environment for production Ethereum client behavior, covering consensus integration, payload validation, and state synchronization. @@ -67,7 +67,7 @@ The `consume engine` command: | Nomenclature | | | -------------- | ------------------ | | Command | `consume rlp` | -| Simulator | `eest/consume-rlp` | +| Simulator | `eels/consume-rlp` | | Fixture format | `blockchain_test` | The RLP consumption method tests execution clients by providing them with RLP-encoded blocks to load upon startup, similar to the block import process during historical synchronization. This method tests the client's core block processing logic without the overhead of network protocols. @@ -103,15 +103,15 @@ The `consume sync` command: ## Engine vs RLP Simulator -The RLP Simulator (`eest/consume-rlp`) and the Engine Simulator (`eest/consume-engine`) should be seen as complimentary to one another. Although they execute the same underlying EVM test cases, the block validation logic is executed via different client code paths (using different [fixture formats](./test_formats/index.md)). Therefore, ideally, **both simulators should be executed for full coverage**. +The RLP Simulator (`eels/consume-rlp`) and the Engine Simulator (`eels/consume-engine`) should be seen as complimentary to one another. Although they execute the same underlying EVM test cases, the block validation logic is executed via different client code paths (using different [fixture formats](./test_formats/index.md)). Therefore, ideally, **both simulators should be executed for full coverage**. ### Code Path Choices -Clients consume fixtures in the `eest/consume-engine` simulator via the Engine API's `EngineNewPayloadv*` endpoint; a natural way to validate, respectively invalidate, block payloads. In this case, there is no flexibility in the choice of code path - it directly harnesses mainnet client functionality. The `eest/consume-rlp` Simulator, however, allows clients more freedom, as the rlp-encoded blocks are imported upon client startup. Clients are recommended to try and hook the block import into the code path used for historical syncing. +Clients consume fixtures in the `eels/consume-engine` simulator via the Engine API's `EngineNewPayloadv*` endpoint; a natural way to validate, respectively invalidate, block payloads. In this case, there is no flexibility in the choice of code path - it directly harnesses mainnet client functionality. The `eels/consume-rlp` Simulator, however, allows clients more freedom, as the rlp-encoded blocks are imported upon client startup. Clients are recommended to try and hook the block import into the code path used for historical syncing. ### Differences -| | `eest/consume-rlp` | `eest/consume-engine` | +| | `eels/consume-rlp` | `eels/consume-engine` | | ----------------------- | ----------------------------------------------------- | ------------------------------------------------------------------ | | **Fixture Format Used** | [`BlockchainTest`](./test_formats/blockchain_test.md) | [`BlockchainTestEngine`](./test_formats/blockchain_test_engine.md) | | **Fork support** | All forks (including pre-merge) | Post-merge forks only (Paris+) | @@ -128,9 +128,9 @@ Clients consume fixtures in the `eest/consume-engine` simulator via the Engine A See [Execute Command](./execute/index.md). -## Two Methods to Run EEST Simulators +## Two Methods to Run EELS Simulators -Many of the methods use the Hive Testing Environment to interact with clients and run tests against them. These methods are also called Hive simulators. While Hive is always necessary to run simulators, they can be called in two different ways. Both of these commands execute the same simulator code, but in different environments, we take the example of the `eest/consume-engine` simulator: +Many of the methods use the Hive Testing Environment to interact with clients and run tests against them. These methods are also called Hive simulators. While Hive is always necessary to run simulators, they can be called in two different ways. Both of these commands execute the same simulator code, but in different environments, we take the example of the `eels/consume-engine` simulator: -1. `./hive --sim=eest/consume-engine` is a standalone command that installs EEST and the `consume` command in a dockerized container managed by Hive. This is the standard method to execute EEST [fixture releases](./releases.md) against clients in CI environments and is the method to generate the results at [hive.ethpandaops.io](https://hive.ethpandaops.io). See [Hive](./hive/index.md) and its [Common Options](./hive/common_options.md) for help with this method. -2. `uv run consume engine` requires the user to clone and [install EEST](../getting_started/installation.md) and start a Hive server in [development mode](./hive/dev_mode.md). In this case, the simulator runs on the native system and communicate to the client via the Hive API. This is particularly useful during test development as fixtures on the local disk can be specified via `--input=fixtures/`. As the simulator runs natively, it is easy to drop into a debugger and inspect the simulator or client container state. See [Hive Developer Mode](./hive/dev_mode.md) for help with this method. +1. `./hive --sim=eels/consume-engine` is a standalone command that installs and configures execution-specs and its `consume` command in a dockerized container managed by Hive. This is the standard method to execute EEST [fixture releases](./releases.md) against clients in CI environments and is the method to generate the results at [hive.ethpandaops.io](https://hive.ethpandaops.io). See [Hive](./hive/index.md) and its [Common Options](./hive/common_options.md) for help with this method. +2. `uv run consume engine` requires the user to clone and [configure execution-specs](../getting_started/installation.md) and start a Hive server in [development mode](./hive/dev_mode.md). In this case, the simulator runs on the native system and communicate to the client via the Hive API. This is particularly useful during test development as fixtures on the local disk can be specified via `--input=fixtures/`. As the simulator runs natively, it is easy to drop into a debugger and inspect the simulator or client container state. See [Hive Developer Mode](./hive/dev_mode.md) for help with this method.