T8N 4844 blob tests state mismatch

[winsvega]

I disabled cpu heavy pyspec test (WARNING: Skipping test_excess_blob_gas because --lowcpu option was specified.) and was able to run the blobs .py tests on nimbus t8n.

Here where I get the stateroot mismatch in 4 tests:

--------
*** TOTAL ERRORS DETECTED: 4 errors during all test execution!
--------
info: (cancun/eip4844_blobs/src/GeneralStateTestsFiller/Pyspecs/cancun/eip4844_blobs/test_blob_txs.py::test_invalid_blob_tx_contract_creation[fork_Cancun-blockchain_test-], fork: Cancun, block: 1)
info: (cancun/eip4844_blobs/src/GeneralStateTestsFiller/Pyspecs/cancun/eip4844_blobs/test_blob_txs.py::test_invalid_tx_blob_count[fork_Cancun-blockchain_test-too_many_blobs], fork: Cancun, block: 1)
info: (cancun/eip4844_blobs/src/GeneralStateTestsFiller/Pyspecs/cancun/eip4844_blobs/test_blob_txs_full.py::test_reject_valid_full_blob_in_block_rlp[fork_Cancun-blockchain_test-one_full_blob_at_the_end_max_txs], fork: Cancun, block: 1)
info: (cancun/eip4844_blobs/src/GeneralStateTestsFiller/Pyspecs/cancun/eip4844_blobs/test_excess_blob_gas_fork_transition.py::test_fork_transition_excess_blob_gas[fork_ShanghaiToCancunAtTime15k-blockchain_test-max_blobs], fork: ShanghaiToCancunAtTime15k, block

This tests are located at BlockchainTests/GeneralStateTests/Pyspec/cancun

Please check if this the t8n only bug or it happen with real nimbus evm

[jangko]

I only get stateroot mismatch from 1st test. While the 2nd and the 4th pass the test. And got exception from the 3rd.

Can you tell me how exactly the test being conducted? or can you provide the input and expected output for the failing cases?

[winsvega]

So basically, I construct env, pre, txs from the tests and pass it to t8n. And then compare the result hashes. But the hash mismatch come from t8n error log in this case.

Also lately get many of these messages

StorageLedger/merge(): Unspecified(Aristo, ctx=mergeFn(), error=UtilsAccWrongStorageRoot)

[winsvega]

I only get stateroot mismatch from 1st test. While the 2nd and the 4th pass the test. And got exception from the 3rd.

Can you tell me how exactly the test being conducted? or can you provide the input and expected output for the failing cases?

This tests are expected to pass. Should I provide the geth vm trace?

[jangko]

I'm more interested in your env and txs, because I might be construct it differently from yours(I don't know if you unpack the txs or not). But geth vm trace also valueable.
So please give me your env, txs and geth vm trace.

[jangko]

Also lately get many of these messages

StorageLedger/merge(): Unspecified(Aristo, ctx=mergeFn(), error=UtilsAccWrongStorageRoot)

It's a known regression, similar to #2228

[winsvega]

what you mean unpack? I just send rlp file for txs

[jangko]

That's what I mean, still in rlp or deserialized into json.

[winsvega]

here is the debug of one of the tests that nimbus is failing with that error
http:https://retesteth.ethdevops.io/temp/nimbus/

[jangko]

Thanks, will look into it

[winsvega]

all fixed now but one

(cancun/eip4844_blobs/src/GeneralStateTestsFiller/Pyspecs/cancun/eip4844_blobs/test_excess_blob_gas_fork_transition.py::test_fork_transition_excess_blob_gas[fork_ShanghaiToCancunAtTime15k-blockchain_test-max_blobs], fork: ShanghaiToCancunAtTime15k, block: 22)

get different state root on block22 perhaps the tranistion. only the stateRoot is different so I think it is related to beaconRoot precompile

[jangko]

@winsvega , my test shows all stateRoot are match for the test you mentioned. Including block 22 and 23.
tested against commit: 8926da0

NAME: src/GeneralStateTestsFiller/Pyspecs/cancun/eip4844_blobs/test_excess_blob_gas_fork_transition.py::test_fork_transition_excess_blob_gas[fork_ShanghaiToCancunAtTime15k-blockchain_test-max_blobs]
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 1, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 2, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 3, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 4, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 5, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 6, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 7, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 8, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 9, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 10, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 11, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 12, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 13, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 14, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 15, 0, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, 0x948622f2177f30e161126ff783f720dced1aa9aaf394c0784c9867b59cf36df9, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 16, 0, 0xa6f5fbf5746b87405d402ba5daa6b8ed95a83a1c374243f76bc0faa1220f4746, 0xa6f5fbf5746b87405d402ba5daa6b8ed95a83a1c374243f76bc0faa1220f4746, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 17, 0, 0x1b76db70e076bb2f6270ab49ac451e9edcfd8238a4e32eef18067254046c0e60, 0x1b76db70e076bb2f6270ab49ac451e9edcfd8238a4e32eef18067254046c0e60, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 18, 0, 0x814ace33f3d42889ca65aa3399c380860d5f9e03661a1db3c5e84f2639b17e9d, 0x814ace33f3d42889ca65aa3399c380860d5f9e03661a1db3c5e84f2639b17e9d, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 19, 0, 0x8ff9d5fc614c0fa3360006eeb598c53dfcc0e16a6723c8df47ade4cab8f491b1, 0x8ff9d5fc614c0fa3360006eeb598c53dfcc0e16a6723c8df47ade4cab8f491b1, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 20, 0, 0x6102f0767e9ac5dc2c0ab9046f5d6ef1443f43d42b04626b460b51a103a43839, 0x6102f0767e9ac5dc2c0ab9046f5d6ef1443f43d42b04626b460b51a103a43839, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 21, 0, 0x8473ad6c1444145a50148ab133f4b95f1c070079cfe9574027415373efd1e97b, 0x8473ad6c1444145a50148ab133f4b95f1c070079cfe9574027415373efd1e97b, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 22, 0, 0x0488e5f75205c63570c7e448aaab098076ebebde078f50f46c3b5216ebaba1f5, 0x0488e5f75205c63570c7e448aaab098076ebebde078f50f46c3b5216ebaba1f5, OK
BlockNumber, EXITCODE, resultStateRoot, expectedStateRoot, status: 23, 0, 0x980e4e4ec883c155b4663eeb6a2983c08744f820f15cca4c1323c46bf3b42775, 0x980e4e4ec883c155b4663eeb6a2983c08744f820f15cca4c1323c46bf3b42775, OK

[winsvega]

Confirm.

Todays errors are around blockhash tests:

http:https://retesteth.ethdevops.io/results/log//2024-06-18-1718697066-nimbus.txt