Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a test runner for the standard ETH blockchain/state validation tests #113

Closed
zah opened this issue Aug 20, 2018 · 5 comments
Closed

Comments

@zah
Copy link
Member

zah commented Aug 20, 2018

The test cases are located here:
https://github.com/ethereum/tests/tree/develop/BlockchainTests
https://github.com/ethereum/tests/tree/develop/GeneralStateTests

... and described here:
https://ethereum-tests.readthedocs.io/en/latest/test_types/blockchain_tests.html
https://ethereum-tests.readthedocs.io/en/latest/test_types/state_tests.html

We need to implement a simple test runner similar to the existing one in test_vm_json.nim. As these tests are significantly larger in size, it will probably be better if add the test suite repo as a git submodule (the CI scripts may have to be modified to fetch all submodules).

Please consider that in the future, these tests will be executable over the JSON-RPC interface using retesteth. The internal APIs can be structured to simplify the implementation of the JSON-RPC procs described here:

ethereum/interfaces#4
ethereum/retesteth#5

@coffeepots
Copy link
Contributor

coffeepots commented Nov 19, 2018

Can this be closed now?

EDIT: Looks like we need RPC interfaces to the tests to close this.

@zah
Copy link
Member Author

zah commented Nov 20, 2018

Well, let's change the closing criteria to "pass all state validation tests"

The test runner has already been implemented here:
https://github.com/status-im/nimbus/blob/master/tests/test_generalstate_json.nim

But a large number of tests are still disabled here:
https://github.com/status-im/nimbus/blob/master/tests/test_generalstate_failing.nim

For newcomers, we've written some helpful notes on debugging test failures in general:
https://github.com/status-im/nimbus/wiki/Understanding-and-debugging-Nimbus-EVM-JSON-tests

I think we'll soon need to update the notes after the debug_traceTransaction implementation is ready and we gain some compatibility with the Remix debugger.

@tersec
Copy link
Contributor

tersec commented Aug 21, 2019

It's getting pretty close now, no?

@jangko
Copy link
Contributor

jangko commented Aug 21, 2019

only two failed GST left

@tersec tersec removed their assignment Nov 13, 2019
@jangko
Copy link
Contributor

jangko commented Feb 19, 2020

fixed by #459, closing

@jangko jangko closed this as completed Feb 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants