Sync fix: GetBlockBodies logic preventing sync, dropping peers
#865
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixes #864 "Sync progress stops at Goerli block 4494913", and equivalent on other networks. The block body fetcher in `blockchain_sync.nim` had an incorrect assumption about how peers respond to `GetBlockBodies`. It was issuing requests for N block bodies and incorrectly handling replies which contained fewer than N bodies. Having received up to 192 headers in a batch, it split the range into smaller `GetBlockBodies` requests, fetched each reply, then combined replies. The effect was Nimbus requested batches of 128+64 block bodies, received gaps in the reply sequence, then aborted. That meant it repeatedly fetched data, then discarded it, and fetched it again, dropping good peers in the process. Aborted and restarted batches occurred with earlier blocks too, but this became more pronounced until there were no suitable peers at batch 4494913..4495104. Here's a trace: ``` TRC 2021-09-29 02:40:24.977+01:00 Requesting block headers file=blockchain_sync.nim:224 start=4494913 count=192 peer=<ENODE> TRC 2021-09-29 02:40:24.977+01:00 >> Sending eth.GetBlockHeaders (0x03) file=protocol_eth65.nim:51 peer=<PEER> startBlock=4494913 max=192 TRC 2021-09-29 02:40:25.005+01:00 << Got reply eth.BlockHeaders (0x04) file=protocol_eth65.nim:51 peer=<PEER> count=192 TRC 2021-09-29 02:40:25.007+01:00 >> Sending eth.GetBlockBodies (0x05) file=protocol_eth65.nim:51 peer=<PEER> count=128 TRC 2021-09-29 02:40:25.209+01:00 << Got reply eth.BlockBodies (0x06) file=protocol_eth65.nim:51 peer=<PEER> count=13 TRC 2021-09-29 02:40:25.210+01:00 >> Sending eth.GetBlockBodies (0x05) file=protocol_eth65.nim:51 peer=<PEER> count=64 TRC 2021-09-29 02:40:25.290+01:00 << Got reply eth.BlockBodies (0x06) file=protocol_eth65.nim:51 peer=<PEER> count=64 WRN 2021-09-29 02:40:25.306+01:00 Bodies len != headers.len file=blockchain_sync.nim:276 bodies=77 headers=192 TRC 2021-09-29 02:40:25.306+01:00 peer disconnected file=blockchain_sync.nim:403 peer=<PEER> TRC 2021-09-29 02:40:25.306+01:00 Finished obtaining blocks file=blockchain_sync.nim:303 peer=<PEER> ``` In practice, for modern peers, Nimbus received shorter replies than it assumed depending on the block sizes on the chain. Geth/Erigon has 2MiB `BlockBodies` response size soft limit. OpenEthereum has 4MiB. Up to Berlin (EIP-2929), Nimbus's fetcher failed often, but there were still some peers serving what Nimbus needed. Just after the start of Berlin, at batch 4494913..4495104 on Goerli, zero peers responded with full size replies for the whole batch, so Nimbus couldn't progress past that point. But there was already a problem happening before that for large blocks, dropping good peers and repeatedly fetching the same block data. Signed-off-by: Jamie Lokier <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #864 "Sync progress stops at Goerli block 4494913", and equivalent on other networks.
The block body fetcher in
blockchain_sync.nimhad an incorrect assumption about how peers respond toGetBlockBodies. It was issuing requests for N block bodies and incorrectly handling replies which contained fewer than N bodies.Having received up to 192 headers in a batch, it split the range into smaller
GetBlockBodiesrequests, fetched each reply, then combined replies. The effect was Nimbus requested batches of 128+64 block bodies, received gaps in the reply sequence, then aborted.That meant it repeatedly fetched data, then discarded it, and fetched it again, dropping good peers in the process.
Aborted and restarted batches occurred with earlier blocks too, but this became more pronounced until there were no suitable peers at batch 4494913..4495104.
Here's a trace:
In practice, for modern peers, Nimbus received shorter replies than it assumed depending on the block sizes on the chain. Geth/Erigon has 2MiB
BlockBodiesresponse size soft limit. OpenEthereum has 4MiB.Up to Berlin (EIP-2929), Nimbus's fetcher failed often, but there were still some peers serving what Nimbus needed.
Just after the start of Berlin, at batch 4494913..4495104 on Goerli, zero peers responded with full size replies for the whole batch, so Nimbus couldn't progress past that point. But there was already a problem happening before that for large blocks, dropping good peers and repeatedly fetching the same block data.