Gas usage consensus error at Mainnet block 6001128

[jlokier]

On Mainnet, sync progress stops at block 6001128. After that point, trace output repeatedly shows gasUsed neq cumulativeGasUsed messages and does not feed blocks to the database, and the peer is disconnected. The gas message means it looks like a consensus logic error.

TRC 2021-09-29 15:13:21.532+01:00 Persisting blocks                  file=persist_blocks.nim:43 fromBlock=6000961 toBlock=6001152
...
DBG 2021-09-29 15:14:35.925+01:00 gasUsed neq cumulativeGasUsed      file=process_block.nim:68 gasUsed=7999726 cumulativeGasUsed=7989726
TRC 2021-09-29 15:14:35.925+01:00 peer disconnected                  file=blockchain_sync.nim:407 peer=<PEER:IP>

• Similar output is seen at many blocks in the range 6001128..6001204.

• Like EIP-170 related consensus error at Goerli block 5080941 #867 "EIP-170 related consensus error at Goerli block 5080941", this is due to behaviour on CREATE or CREATE2. But it's a different cause, and this message doesn't mention EIP-170.

• This was fixed accidentally by a fix for Goerli block 5080941 which refactored the logic in writeContract. Because of this overlap, these two issues were thought to be the same bug at first.

• But it wasn't fixed by the other fix for Goerli block 5080941 in commit 6548ff9 "fixes CREATE/CREATE2's returndata bug", which changes handling of received returnData from calling a nested contract instead of changing writeContract.

• This is how we know there's more going on.

• From the diffences between those two patches, we can see the issue is connected to SELFDESTRUCT interaction with CREATE or CREATE2, in writeContract.