Releases · ethereum/execution-spec-tests

@fselmo

What's Changed

The first release (bal@v1.0.0) revealed some updates needed for EIP-7928: Block-Level Access Lists specifications and tests. This patch release includes fixes to both the specifications (EELS) and the tests (EEST), outlined below.

Fixes

EELS

The specification was missing the state tracker on both the beacon roots and history storage pre-execution system contract calls in the t8n setup, so these were missing from the BAL. Some unit tests were also updated for consistency though they didn't affect the test vectors.

fix(bal): Initialize the state tracker before system contract calls by @fselmo in ethereum/execution-specs@c730e6d
correct system contract addresses by @nerolation in ethereum/execution-specs@70c0f9e

EEST

The testing side had all invalid test cases misconfigured due to some model changes in the repository. The valid test cases did not change in setup but now include the previously-missing contract calls from the updated spec changes above.

fix(tests): Block-Level Access List invalid tests by @fselmo in #2121

EESTs Fusaka Devnet 5 release v2.1.0.

Summary

Small patch on top of the latest Fusaka Devnet-5 release including a fix for some hive test exceptions.

Included EIPs

The list below links the specific commit versions of the EIPs included in this release:

Emoji Key

✋ - No change since last release
🆕 - Brand new EIP tests added
🆙 - Existing EIP tests updated

@fselmo

EEST feature release for EIP-7928: Block-Level Access Lists tests v1.0.0.

⚠️ Important: This release revealed updates needed on both the specifications and the testing side. Look for an updated bal release with corrected and updated test vectors.

Summary

This is the initial tests release for block-level access lists support for Glamsterdam.
Tests in this release are filled only for the initial test cases related to Block-Level Access list, which only exist in the Amsterdam fork.

Relevant changes

feat(tests): Support for EIP-7928 - Block-Level Access Lists by @fselmo in #2067
fix(tests): Restrict extra args being passed to BaseTest; fix valid BAL tests by @fselmo in #2102

Built on top of initial work done by @raxhvl and @nerolation.

🇯🇵 Summary

EEST Fujisan is our first full release for Osaka, the first full release since Pectra!

In addition to the latest Osaka specific test cases, it includes re-filled GeneralStateTests from ethereum/tests (now fully maintained within EEST under tests/static) for Osaka adhering to the transaction gas limit cap from EIP-7825. Further framework changes include new simulators, test formats and test types.

⚔️ Future Weld with EELS

EEST will merge with EELS during Q4 2025, after which EEST becomes read-only for external contributors.

What this means?

All EEST code moves to the EELS repository.
New EEST framework location: execution-specs/src/ethereum_spec_tests/.
New EEST tests location: execution-specs/tests/eest/.
Future PRs go to EELS instead of EEST.

Important Notes

All PRs for tests and framework changes should still be directed at EEST until further notice.
There will be a brief freeze on EEST contributions during Q4 "The Switch", after which contributors can continue as before, but in EELS.
Test releases will continue from EEST as normal before, during, and after this transition.

More information will be communicated accordingly through the normal communication channels.

❗Current Status Quo

Test Fixtures Overview

fixtures_static.tar.gz has been deprecated.
fixtures_stable.tar.gz & fixtures_develop.tar.gz now both contain re-filled static tests, GeneralStateTests from ethereum/tests, filled from Cancun.
fixtures_stable.tar.gz contains tests filled for forks until Prague.
fixtures_develop.tar.gz contains tests filled for forks until Osaka.
fixtures_benchmark.tar.gz contains benchmark tests filled for only Prague.

EL Client Test Requirements

Prague Coverage (Mainnet):

Run all state_test's & blockchain_test's from fixtures_stable.tar.gz.
Run only BlockchainTests from the latest ethereum/tests release, filled until Prague.

Fusaka Coverage (Including Mainnet):

Run all state_test's & blockchain_test's from fixtures_develop.tar.gz
Run only BlockchainTests from the latest ethereum/tests release, filled until Prague.

Note: If you require GeneralStateTests from ethereum/tests filled for forks before Cancun then you must get these from the latest ethereum/tests release.

Benchmark Tests

For the most up-to-date benchmark tests, use fixtures_benchmark.tar.gz.

Note: Benchmark tests for Osaka (compliant with EIP-7825 transaction gas limit cap) will be added in a future feature release.

Test Fixture Formats

This release includes 2 new test formats designed primarily for Hive simulators:

blockchain_tests_engine_x: An optimized version of blockchain_tests_engine where multiple tests share the same genesis state, allowing multiple tests to run on a single client instantiation within Hive's consume-engine. The standard format requires a fresh client startup for each test. Due its combined genesis state, this is additionally the primary format used by the Nethermind team for benchmarking.
blockchain_tests_sync: A new format adjacent to the existing blockchain_tests_engine format. Used specifically for the upcoming consume-sync simulator, which delivers engine payloads from test fixtures to the client under test, then sync's a separate client to it. This test fixture is only marked to be filled for the EIP-7934 block RLP limit tests in Osaka.

Tooling & Simulators

Improved tooling and new Hive simulators are additionally included in this release:

execute remote: this command now supports optional Engine RPC endpoints (--engine-endpoint) with JWT authentication #2070.
- This allows manual control over block creation and transaction inclusion for more deterministic test execution on live networks. Previously, execute remote could only submit transactions and rely on the network's automatic block production, but now it can actively drive chain progression by creating blocks on-demand via the Engine API.
consume sync: Adjacent to consume-engine, designed to work with the blockchain_tests_sync format for testing client sync scenarios.
execute blobs: A new Hive specific simulator that uses the EEST execute pytest plugin. Sends blob transactions to the client under test and verifies its engine_getBlobsVX endpoint. Requires tests to be written with a new python test format blob_transaction_test. Primarily used to test PeerDAS from the EL perspective.
execute eth config: A command used to test the eth_config endpoint from EIP-7910. Can be ran remotely or within Hive.

Filling For Stateless Clients

A witness-filler extension is included in this release, allowing for tests to be filled that include an executionWitness for each fixture #2066. This essentially calls an external executable written in rust, and hence must be installed for usage within fill using the --witness flag. The current approach is below:

cargo install --git https://github.com/kevaundray/reth.git --branch jsign-witness-filler witness-filler
uv run fill ... --output=fixtures-witness --witness --clean

Note: The witness-filler executable is not maintained by EEST so we cannot help with any issues.

💥 Breaking Changes

Important changes for EEST superusers

EEST now requires uv>=0.7.0 (#1904). If your version of uv is too old.
When filling fixtures transition forks are included within there respective "to" fork, where --fork Osaka will now include PragueToOsakaAtTime15k. Previously transitions fork would only be included when filling with --from Prague --until Osaka flags.
Python 3.10 support was removed in this release (#1808).
EEST no longer allows usage of Yul code in Python tests. From now on, please make use of our opcode wrapper. Yul code is now only allowed in the "static tests" located in ./tests/static/ (these are test cases defined by JSON and YAML files instead of Python test functions that were originally maintained in ethereum/tests).
In order to fill the static tests (which is not the case by default), please ensure that solc is located in your PATH.
The output behavior of fill has changed (#1608):
- Before: fill wrote fixtures into the directory specified by the --output flag (default: fixtures). This could have many unintended consequences, including unexpected errors if old or invalid fixtures existed in the directory (for details see #1030).
- Now: fill will exit without filling any tests if the specified directory exists and is not-empty. This may be overridden by adding the --clean flag, which will first remove the specified directory.
Writing debugging information to the EVM "dump directory" by default has been disabled. To obtain debug output, the --evm-dump-dir flag must now be explicitly set. As a consequence, the now redundant --skip-evm-dump option was removed (#1874). This undoes functionality originally introduced in #999 and #1150.

Feature `zkevm` updated to `benchmark`

Due to the crossover between zkevm and benchmark tests, all instances of the former have been replaced with the latter nomenclature. Repository PR labels and titles are additionally updated to reflect this change.

This update renames the zkevm feature release to benchmark and further expands the latter for 1M, 10M, 30M, 45M, 60M, 90M, and 120M block gas limits in fixtures_benchmark.tar.gz.

To select a test for a given gas limit, the IDs of the tests have been expanded to contain benchmark-gas-value_XM, where X can be any of the aforementioned values.

The benchmark release also now includes BlockchainEngineX format that combines most of the tests into a minimal amount of genesis files. For more info see Blockchain Engine X Tests in the EEST documentation.

Users can select any of the artifacts depending on their benchmarking or testing needs for their provers.

🔑 Other Key Changes

🛠️ Framework

🔀 Refactoring

🔀 Move TransactionType enum from test file to proper module location in ethereum_test_types.transaction_types for better code organization ...

EESTs Fusaka Devnet 5 release v2.0.0.

Summary

Builds on top of the last Fusaka Devnet-5 release, contains additional coverage for EIP-7823 (#2058). Adds extra tests for BPOs regarding the reserve mechanism for EIP-7918 (#2074), specifically when the update fraction is reduced alongside the max and target blobs of the blob schedule - the past tests only focused on the increase.

All tests are filled from Osaka until BPO4 in this release. Prague fixtures are removed from this release as requested. The will be included within the next v5.0.0 release.

Included EIPs

The list below links the specific commit versions of the EIPs included in this release:

Emoji Key

✋ - No change since last release
🆕 - Brand new EIP tests added
🆙 - Existing EIP tests updated

@yperbasis

EESTs Fusaka Devnet 5 release v1.1.0.

Summary

Builds on top of the last Fusaka Devnet-5 release, contains additional coverage for EIP-7883 (#1929) & EIP-7825 (#2034). Fixes an issue (found by @yperbasis) with an EIP-4844 transition test that used the EIP-7918 reserve price activation (#2059).

All tests are filled from Prague until Osaka in this release.

Included EIPs

The list below links the specific commit versions of the EIPs included in this release:

Emoji Key

✋ - No change since last release
🆕 - Brand new EIP tests added
🆙 - Existing EIP tests updated

EESTs Fusaka Devnet 5 release v1.0.0.

Summary

Builds on top of the last Fusaka Devnet-3 release, containing additional test cases for EIP-7883 & EIP-7823. Initial tests for BPOs are added for the bug caught on fusaka-devnet-4, with regards to the reserve price activation in EIP-7918 on the BPO transition block (#2035).

All existing EEST tests are filled from Prague (including Osaka) in this release providing coverage from a stable mainnet basis!

Included EIPs

The list below links the specific commit versions of the EIPs included in this release:

Emoji Key

✋ - No change since last release
🆕 - Brand new EIP tests added
🆙 - Existing EIP tests updated

@marioevz

Benchmark Tests

Contains consolidated blockchain tests that can be used for benchmarking, as well as the original blockchain_tests and blockchain_tests_engine, but deprecates state_test format (only for benchmark releases, this format will still be available for normal releases).

This release also deprecates Cancun fork and contains only Prague.

Single consolidated genesis

This release contains a single genesis file for the Engine X format. For every single test contained in this release, only one genesis file has to be loaded and, theoretically, all tests should run with a single client instance without having to restart at any point.

For more information on the new format see Blockchain Engine X Tests in the EEST documentation.

Forks Included

Prague
~~Cancun~~ (Deprecated)

Note: Osaka is still not supported for benchmarks, but will be in a future release.

What's Changed

fix(github): Remove --solc-version flag from fixture building action by @marioevz in #1937
fix(docs): fix broken link that leads to auto-issue generation #1930 by @felix314159 in #1938
(feat)ci: pypy3.11 support by @kclowes in #1854
fix(plugins): Remove cwd change before running pytest by @marioevz in #1943
fix(clis/besu): blob count max exception mapper by @Gabriel-Trintinalia in #1944
feat(tests): remove Blake2b max rounds static test by @chfast in #1941
feat(tests): add eip7702 test case for clz opcode by @LouisTsai-Csie in #1853
fix(mypy): adds mypy ignore comments and add ignore rule for imports by @felix314159 in #1951
chore: bump eels resolver by @spencer-tb in #1950
fix(plugins): Don't copy test list during collection by @marioevz in #1936
chore(tests): remove slow/zkevm markers from benchmark tests by @spencer-tb in #1953
fix(unit_tests): fixed blob unit tests when run with xdist by @felix314159 in #1946
feat(tests): remove the static performance test calling SHA256 5000x by @chfast in #1949
fix(static_tests): adjusted gasLimit to only allow gasLimit of up to 30mil by @felix314159 in #1924
feat(consume): add --extract-to parameter for direct fixture extraction by @richardgreg in #1861
fix(tests): EIP-7825: Pre-Osaka tests consuming tx gas above cap by @marioevz in #1928
chore(benchmark): don't fill benchmark tests by default by @spencer-tb in #1920
chore(clis): update geth exception mapper for TYPE_3_TX_BLOB_COUNT_EXCEEDED by @MariusVanDerWijden in #1958
chore(docs): add benchmark marker to test case reference by @spencer-tb in #1960
feat(specs,tests): Update all benchmark tests to use gas_benchmark_value by @marioevz in #1935
chore(ci): bump to use latest eels forks/osaka by @spencer-tb in #1962
feat(tests): extra clz param for scenarios test case by @spencer-tb in #1965
chore(fill): remove fill warnings by @spencer-tb in #1964
feat(tests): add call memory expansion with no return size test by @JereSalo in #1952
feat(cli): resolve updated execution_spec exceptions by @souradeep-das in #1784
tests(benchmark): expand worst case modexp benchmarks by @LouisTsai-Csie in #1780
chore(tests|forks): add max blobs per tx limit by @spencer-tb in #1884
fix(fill): allow lowercase fork declaration in fill command by @felix314159 in #1969
feat(tests): add initcode context extra test for clz opcode by @LouisTsai-Csie in #1831
chore(ci|cli): improve coverage ci and compare fixtures python script by @spencer-tb in #1970
chore(clis): add erigon exception mapper for TYPE_3_TX_BLOB_COUNT_EXCEEDED by @yperbasis in #1974
feat(tests): add call operation test case for CLZ by @LouisTsai-Csie in #1829
feat(tests): extra stack operation test for CLZ opcode by @LouisTsai-Csie in #1821
refactor(specs/static_state): Fork string parsing by @marioevz in #1977
fix(consume): fixes issue where nethtest would not be recognized as valid t8n on certain systems by @felix314159 in #1973
feat(benchmark): add clz opcode test case by @LouisTsai-Csie in #1845
fix(tests): Issues with rlp block limit tests and --generate-pre-alloc-groups by @fselmo in #1989
feat(ci): Build releases in self-hosted-ghr, fetch solidity, lllc from github releases by @marioevz in #1990
test: add max block size test using access lists by @nerolation in #1932
feat(checklist): Add Modified Transaction Validity Constraint by @marioevz in #1988
chore(tests): fix pre alloc grouping remaining fails by @spencer-tb in #1975
fix(tests/static): Fix all static tests for Osaka fork by @marioevz in #1980
feat(test): add test for memory expansion on early revert in CALL by @JereSalo in #1967
chore(tests): remove temp fix for block rlp type 4 by @spencer-tb in #1995
chore: align gas limit with mainnet by @LouisTsai-Csie in #1994
chore(benchmark): update gas limit configuration by @LouisTsai-Csie in #1983
fix(github): Set x-dist to auto for all evm impl by @marioevz in #1996
chore(tests): update max blob per tx test errors by @spencer-tb in #1997
chore(ci): preparation for next full release by @spencer-tb in #1954
feat(fill): Disable --generate-all-formats for static tests by @marioevz in #1998
feat(tests): add critical BLS tests with invalid point coordinates by @chfast in #2000
feat(tests): extra BLS tests with invalid point coords by @spencer-tb in #2004
feat(tests): add case for large exp with zero mod/base length by @spencer-tb in #2005
chore(benchmark): update max env gas by @spencer-tb in #2001
fix(tests): Fix ModExpInput.from_bytes by @marioevz in #1999
test(osaka): add edge case test vectors for EIP-7883 MODEXP gas calculation by @bshastry in #1993
refactor(fill): remove spaces from test ids/fixture names by @LouisTsai-Csie in #2009
chore(tests): exception hotfix by @spencer-tb in #2011
bug(cli): fixed bug where --trace would never be enabled for geth's evm by @felix314159 in #2012
feat(tests): Test 256verify modular comparison by @kclowes in #2015
feat(specs,filler,ci): Merge static tests into stable/develop, make --generate-all-formats compatible with static tests by @marioevz in #2006

New Contributors

@Gabriel-Trintinalia made their first contribution in #1944
@souradeep-das made their first contribution in #1784
@nerolation made their first contribution in #1932
@bshastry made their first contribution in #1993

Full Changelog: https://github.com/ethereum...

Benchmark Tests

Contains consolidated blockchain tests that can be used for benchmarking, as well as the original state_tests, blockchain_tests and blockchain_tests_engine.

The included file fixtures_benchmark.tar.gz contains all different benchmark test files for multiple gas limits: 1M, 10M, 30M, 45M, 60M, 90M, and 120M.

To select a test for a given gas limit, the IDs of the tests have been expanded to contain benchmark-gas-value_XM, where X can be any of the aforementioned values (e.g. benchmark-gas-value_1M, benchmark-gas-value_10M, and so on).

For more information on the new format see Blockchain Engine X Tests in the EEST documentation.

Forks Included

Cancun
Prague

Note: Osaka is still not supported for benchmarks, but will be in a future release.

Full Changelog: https://github.com/ethereum/execution-spec-tests/compare/benchmark@v0.0.2...benchmark@v0.0.3

EESTs Fusaka Devnet 3 release v1.0.0.

❌ EIP-7907 is removed from Fusaka! ❌

Summary

Builds on top of the last Fusaka Devnet-2 release, containing additional test cases for EIP-7934 & EIP-7939, removing EIP-7907 test cases although still includes re-filled Shanghai initcode tests to provide some validation for the removal of EIP-7907 from clients.

New tests for the maxBlobsPerTx limit are added for EIP-7594 including more modified EIP-4844 tests.

Included EIPs

The list below links the specific commit versions of the EIPs included in this release:

EIP-7823: Set upper bounds for MODEXP ✋
EIP-7883: ModExp Gas Cost Increase 🆙 (gas price changes)
EIP-7918: Blob base fee bounded by execution cost 🆙 (blob base fee cost changed)
EIP-7825: Transaction Gas Limit Cap 🆙 (changed from 30M to 2^24)
EIP-7951: Precompile for secp256r1 Curve Support 🆙 (gas price changes)
EIP-7934: RLP Execution Block Size Limit ✋
EIP-7939: Count leading zeros (CLZ) opcode 🆙 (gas price changes)

Emoji Key

✋ - No change since last release
🆕 - Brand new EIP tests added
🆙 - Existing EIP tests updated

Releases: ethereum/execution-spec-tests

bal@v1.0.1

What's Changed

Fixes

EELS

EEST

Contributors

Uh oh!

fusaka-devnet-5@v2.1.0

Summary

Included EIPs

Emoji Key

Uh oh!

bal@v1.0.0

Summary

Relevant changes

Contributors

Uh oh!

v5.0.0

🇯🇵 Summary

⚔️ Future Weld with EELS

❗Current Status Quo

Test Fixtures Overview

EL Client Test Requirements

Benchmark Tests

Test Fixture Formats

Tooling & Simulators

Filling For Stateless Clients

💥 Breaking Changes

Important changes for EEST superusers

Feature zkevm updated to benchmark

🔑 Other Key Changes

🛠️ Framework

🔀 Refactoring

Contributors

Uh oh!

fusaka-devnet-5@v2.0.0

Summary

Included EIPs

Emoji Key

Uh oh!

fusaka-devnet-5@v1.1.0

Summary

Included EIPs

Emoji Key

Contributors

Uh oh!

fusaka-devnet-5@v1.0.0

Summary

Included EIPs

Emoji Key

Uh oh!

benchmark@v0.0.4

Benchmark Tests

Single consolidated genesis

Forks Included

What's Changed

New Contributors

Contributors

Uh oh!

benchmark@v0.0.3

Benchmark Tests

Forks Included

Uh oh!

fusaka-devnet-3@v1.0.0

Summary

Included EIPs

Emoji Key

Uh oh!

Feature `zkevm` updated to `benchmark`