Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/filling_tests/test_ids.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@ The test framework can also generate blockchain tests containing blocks that spa

Each Python test case is also typically parametrized by test type, respectively fixture format. For example, if the test is implemented as a `state_test`, the test framework will additionally generate the following blockchain test fixtures (consisting of a single block with a single transaction):

- a `blockchain_test` which can be tested via the Hive `eest/consume-rlp` simulator (or directly via a dedicated client interface).
- a `blockchain_engine_test` (for post-merge forks) which can be tested via the Hive `eest/consume-engine` simulator.
- a `blockchain_test` which can be tested via the Hive `eels/consume-rlp` simulator (or directly via a dedicated client interface).
- a `blockchain_engine_test` (for post-merge forks) which can be tested via the Hive `eels/consume-engine` simulator.

### Example: The Test IDs generated for `test_chainid`

Expand Down
2 changes: 1 addition & 1 deletion docs/running_tests/consume/exceptions.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ uv run consume engine --disable-strict-exception-matching=nimbus-el
Enable verbose client output:

```bash
./hive --sim ethereum/eest/consume-engine \
./hive --sim ethereum/eels/consume-engine \
--docker.output \
--sim.loglevel 5
```
Expand Down
4 changes: 2 additions & 2 deletions docs/running_tests/execute/hive.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@

Tests can be executed on a local hive-controlled single-client network by running the `execute hive` command.

## The `eest/execute-blobs` Simulator
## The `eels/execute-blobs` Simulator

The `blob_transaction_test` execute test spec sends blob transactions to a running client in order to verify its `engine_getBlobsVX` endpoint behavior. These tests can be run using:

```bash
./hive --client besu --client-file ./configs/osaka.yaml --sim ethereum/eest/execute-blobs
./hive --client besu --client-file ./configs/osaka.yaml --sim ethereum/eels/execute-blobs
```

See [Hive](../hive/index.md) for help installing and configuring Hive.
Expand Down
6 changes: 3 additions & 3 deletions docs/running_tests/hive/client_config.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,17 +102,17 @@ cp -r /path/to/your/go-ethereum ./clients/go-ethereum/go-ethereum-local
Force rebuild base images:

```bash
./hive --docker.pull --sim ethereum/eest/consume-engine
./hive --docker.pull --sim ethereum/eels/consume-engine
```

Force rebuild specific client:

```bash
./hive --docker.nocache "clients/go-ethereum" --sim ethereum/eest/consume-engine
./hive --docker.nocache "clients/go-ethereum" --sim ethereum/eels/consume-engine
```

Show the docker container build output:

```bash
./hive --docker.buildoutput --sim ethereum/eest/consume-engine
./hive --docker.buildoutput --sim ethereum/eels/consume-engine
```
18 changes: 9 additions & 9 deletions docs/running_tests/hive/common_options.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
# Common Simulator Options

All EEST Hive simulators share common command-line options and patterns.
All execution-specs (EELS) Hive simulators share common command-line options and patterns.

## Basic Usage

While they may be omitted, it's recommended to specify the `fixtures` and `branch` simulator build arguments when running EEST simulators.
While they may be omitted, it's recommended to specify the `fixtures` and `branch` simulator build arguments when running execution-specs simulators.

For example, this runs "stable" fixtures from the v4.3.0 [latest stable release](../releases.md#standard-releases) and builds the simulator at the v4.3.0 tag:

```bash
./hive --sim ethereum/eest/consume-engine \
./hive --sim ethereum/eels/consume-engine \
--sim.buildarg fixtures=stable@v4.3.0 \
--sim.buildarg branch=v4.3.0 \
--client go-ethereum
Expand All @@ -20,15 +20,15 @@ For example, this runs "stable" fixtures from the v4.3.0 [latest stable release]
Run a subset of tests by filtering tests using `--sim.limit=<regex>` to perform a regular expression match against test IDs:

```bash
./hive --sim ethereum/eest/consume-engine --sim.limit ".*eip4844.*"
./hive --sim ethereum/eels/consume-engine --sim.limit ".*eip4844.*"
```

### Collect Only/Dry-Run

The `collectonly:` prefix can be used to inspect which tests would match an expression (dry-run), `--docker.output` must be specified to see the simulator's collection result:

```bash
./hive --sim ethereum/eest/consume-engine \
./hive --sim ethereum/eels/consume-engine \
--sim.buildarg fixtures=stable@v4.3.0 \
--sim.buildarg branch=v4.3.0 \
--docker.output \
Expand All @@ -40,7 +40,7 @@ The `collectonly:` prefix can be used to inspect which tests would match an expr
The `id:` prefix can be used to select a single test via its ID (this will automatically escape any special characters in the test case ID):

```console
./hive --sim ethereum/eest/consume-engine \
./hive --sim ethereum/eels/consume-engine \
--sim.buildarg fixtures=stable@v4.3.0 \
--sim.buildarg branch=v4.3.0 \
--docker.output \
Expand All @@ -52,21 +52,21 @@ The `id:` prefix can be used to select a single test via its ID (this will autom
To run multiple tests in parallel, use `--sim.parallelism`:

```bash
./hive --sim ethereum/eest/consume-rlp --sim.parallelism 4
./hive --sim ethereum/eels/consume-rlp --sim.parallelism 4
```

### Output Options

See hive log output in the console:

```bash
./hive --sim ethereum/eest/consume-engine --sim.loglevel 5
./hive --sim ethereum/eels/consume-engine --sim.loglevel 5
```

### Container Issues

Increase client timeout:

```bash
./hive --client.checktimelimit=180s --sim ethereum/eest/consume-engine
./hive --client.checktimelimit=180s --sim ethereum/eels/consume-engine
```
43 changes: 21 additions & 22 deletions docs/running_tests/hive/dev_mode.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Hive Development Mode

This section explains how to run EEST simulators using their EEST commands, e.g., `uv run consume engine`, against a Hive "development" server as apposed to using the standalone `./hive` command.
This section explains how to run EELS simulators using their Python-based commands, e.g., `uv run consume engine`, against a Hive "development" server as apposed to using the standalone `./hive` command.

This avoids running the simulator in a dockerized environment and has several advantages:

Expand All @@ -18,7 +18,7 @@ This avoids running the simulator in a dockerized environment and has several ad

### Prerequisites

- EEST is installed, see [Installation](../../getting_started/installation.md)
- The execution-specs repo is setup in development mode, see [Installation](../../getting_started/installation.md)
- Hive is built, see [Hive](../hive/index.md#quick-start).

## Hive Dev Setup on Linux
Expand All @@ -29,7 +29,7 @@ This avoids running the simulator in a dockerized environment and has several ad
./hive --dev --client go-ethereum --client-file clients.yaml --docker.output
```

2. In a separate shell, configure environment for EEST:
2. In a separate shell, configure environment for execution-specs:

=== "bash/zsh"

Expand All @@ -43,7 +43,7 @@ This avoids running the simulator in a dockerized environment and has several ad
set -x HIVE_SIMULATOR http://127.0.0.1:3000
```

3. Run EEST consume commands
3. Run execution-specs `consume` commands

```bash
uv run consume engine --input ./fixtures -k "test_chainid"
Expand All @@ -56,37 +56,36 @@ Due to Docker running within a VM on macOS, the host machine and Docker containe

1. Linux VM: Run a Linux virtual machine on your macOS and execute the standard development workflow above from within the VM.
2. Remote Linux: SSH into a remote Linux environment (server or cloud instance) and run development mode there.
3. **Docker Development Image**: Create a containerized EEST environment that runs within Docker's network namespace (recommended).
3. **Docker Development Image**: Create a containerized execution-specs environment that runs within Docker's network namespace (recommended).

The following section details the setup and usage of option 3.

### EEST Docker Development Image
### EELS Docker Development Image

Within the [`eest/`](https://github.com/ethereum/hive/tree/master/simulators/ethereum/eest) directory of hive, a new dockerfile must be created: `Dockerfile.dev`, with the following contents:
Within the [`eels/`](https://github.com/ethereum/hive/tree/master/simulators/ethereum/eels) directory of hive, a new dockerfile must be created: `Dockerfile.dev`, with the following contents:

```docker
FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim
ARG branch=main
ENV GIT_REF=${branch}
ARG branch=""

RUN apt-get update && apt-get install -y git
RUN git init execution-spec-tests && \
cd execution-spec-tests && \
git remote add origin https://github.com/ethereum/execution-spec-tests.git && \
git fetch --depth 1 origin $GIT_REF && \
git checkout FETCH_HEAD;

WORKDIR /execution-spec-tests
RUN git clone --depth 1 https://github.com/ethereum/execution-specs.git && \
cd execution-specs && \
if [ -n "$branch" ]; then \
git fetch --depth 1 origin "$branch" && \
git checkout FETCH_HEAD; \
fi
WORKDIR /execution-specs/packages/testing
RUN uv sync
ENTRYPOINT ["/bin/bash"]
```

This dockerfile will be our entry point for running EEST commands.
This dockerfile will be our entry point for running simulator commands.

### `eest/` Hive Directory Structure
### `eels/` Hive Directory Structure

```tree
├── eest
├── eels
│ ├── Dockerfile.dev
│ ├── consume-block-rlp
│ │ └── Dockerfile
Expand All @@ -108,10 +107,10 @@ This dockerfile will be our entry point for running EEST commands.
./hive --dev --dev.addr <LOCAL_IP>:3000 --client go-ethereum --client-file clients.yaml
```

3. In a separate terminal session, build the EEST development image:
3. In a separate terminal session, build the EELS development image:

```bash
cd simulators/ethereum/eest/
cd simulators/ethereum/eels/
docker build -t macos-consume-dev -f Dockerfile.dev .
```

Expand All @@ -136,7 +135,7 @@ When Hive runs in dev mode:
3. Keeps the Hive Proxy container running between test executions.
4. Waits for external simulator connections via the API.

This allows EEST's consume commands to connect to the running Hive instance and execute tests interactively.
This allows the EELS's consume commands to connect to the running Hive instance and execute tests interactively.

## More Options Available

Expand Down
2 changes: 1 addition & 1 deletion docs/running_tests/hive/index.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Hive

@ethereum/hive is a containerized testing framework that helps orchestrate test execution against Ethereum clients. Hive is incredibly extensible; new test suites can be implemented in a module manner as "simulators" that interact with clients to test certain aspects of their behavior. EEST implements several simulators, see [Running Tests](../running.md) for an overview.
@ethereum/hive is a containerized testing framework that helps orchestrate test execution against Ethereum clients. Hive is incredibly extensible; new test suites can be implemented in a module manner as "simulators" that interact with clients to test certain aspects of their behavior. The execution-specs `testing` package implements several simulators, see [Running Tests](../running.md) for an overview.

## Quick Start

Expand Down
6 changes: 3 additions & 3 deletions docs/running_tests/releases.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@
| Format | Consumed by the client | Location in `.tar.gz` release |
| -------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| [State Tests](./test_formats/state_test.md) | - directly via a `statetest`-like command<br/> (e.g., [go-ethereum/cmd/evm/staterunner.go](https://github.com/ethereum/go-ethereum/blob/4bb097b7ffc32256791e55ff16ca50ef83c4609b/cmd/evm/staterunner.go)) | `./fixtures/state_tests/` |
| [Blockchain Tests](./test_formats/blockchain_test.md) | - directly via a `blocktest`-like command<br/> (e.g., [go-ethereum/cmd/evm/blockrunner.go](https://github.com/ethereum/go-ethereum/blob/4bb097b7ffc32256791e55ff16ca50ef83c4609b/cmd/evm/blockrunner.go))</br>- using the [RLPeest/consume-rlp Simulator](./running.md#rlp) via block import | `./fixtures/blockchain_tests/` |
| [Blockchain Engine Tests](./test_formats/blockchain_test_engine.md) | - using the [eest/consume-engine Simulator](./running.md#engine) and the Engine API | `./fixtures/blockchain_tests_engine/` |
| [Blockchain Tests](./test_formats/blockchain_test.md) | - directly via a `blocktest`-like command<br/> (e.g., [go-ethereum/cmd/evm/blockrunner.go](https://github.com/ethereum/go-ethereum/blob/4bb097b7ffc32256791e55ff16ca50ef83c4609b/cmd/evm/blockrunner.go))</br>- using the [eels/consume-rlp Simulator](./running.md#rlp) via block import | `./fixtures/blockchain_tests/` |
| [Blockchain Engine Tests](./test_formats/blockchain_test_engine.md) | - using the [eels/consume-engine Simulator](./running.md#engine) and the Engine API | `./fixtures/blockchain_tests_engine/` |
| [Transaction Tests](./test_formats/transaction_test.md) | - using a new simulator coming soon | None; executed directly from Python source,</br>using a release tag |
| Blob Transaction Tests | - using the [eest/execute-blobs Simulator](./execute/hive.md#the-eestexecute-blobs-simulator) and | None; executed directly from Python source,</br>using a release tag |
| Blob Transaction Tests | - using the [eels/execute-blobs Simulator](./execute/hive.md#the-eestexecute-blobs-simulator) and | None; executed directly from Python source,</br>using a release tag |

## Release URLs and Tarballs

Expand Down
Loading
Loading