Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[R4R] Pipecommit enable trie prefetcher #992

Merged
merged 8 commits into from
Jul 29, 2022
Merged

[R4R] Pipecommit enable trie prefetcher #992

merged 8 commits into from
Jul 29, 2022

Conversation

qinglin89
Copy link
Contributor

@qinglin89 qinglin89 commented Jul 14, 2022 •
edited
Loading

Description

current pipecommit mode would use dummyRoot as account storageRoot before previous block had done commit. This would lead trirePrefetcher not able to work.

Rationale

Remove dummyRoot to keep previous storage root while doing triePrefetch and recover stateRoot to enable effective triePrefetch in pipecommit mode
It's fine to do triePrefetch on previous stateRoot/storageRoot and would do a lot help for all the accounts/storages which are not modified during previous block.
And this would cause no harm since we use the corrected root when we do updateTrie and commit, which would ignore the stale node prefetched and the path node in cache would still help even though they were prefetched from a stale root.
tell us why we need these changes...

Test

metrics on verifystate stage under pipecommit mode

verifystate-metrics

metrics

green-line: before
yellow-line: after

triePrefetch efficiency:

  • on Account   up from 0% to 100%

    • before: 0%
    accountPrefetch_before
    • after: 100%
    accountPrefetch_after

  • on Storage

    • before: about 70%
    storagePrefetch_before
    • after: about 84%
    storagePrefetch_after

Changes

Notable changes:

  • remove dummyroot, keep root before previous block for triePrefetch on storage
  • use stateRoot of block before previous one for triePrefetchon account

@qinglin89 qinglin89 closed this Jul 15, 2022
@qinglin89 qinglin89 reopened this Jul 21, 2022
@qinglin89 qinglin89 changed the title [WIP] Pipecommit enabletrieprefetcher [R4R] Pipecommit enabletrieprefetcher Jul 22, 2022
@qinglin89 qinglin89 force-pushed the pipecommit-enabletrieprefetcher branch from 9d47a7d to ae14c5c Compare July 22, 2022 10:40
brilliant-lx
brilliant-lx reviewed Jul 23, 2022
View reviewed changes
Copy link
Collaborator

@brilliant-lx brilliant-lx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To support the account stale metrics, this PR change some API interface, e.g. Snapshot.Parent, NewTriePrefetch... It may not worth it.
Pls consider remove metrics in this PR.

@qinglin89
Copy link
Contributor Author

To support the account stale metrics, this PR change some API interface, e.g. Snapshot.Parent, NewTriePrefetch... It may not worth it. Pls consider remove metrics in this PR.

These modify was not meant to support the metrics, they just used for account root prefetch in pipecommit mode, and the new metrics was added for consistence since new account root was added for prefetch.

@qinglin89
Copy link
Contributor Author

@hackmod

any comments ？

brilliant-lx
brilliant-lx reviewed Jul 26, 2022
View reviewed changes
core/state/trie_prefetcher.go Outdated
fetchers map[common.Hash]*subfetcher // Subfetchers for each trie
db Database // Database to fetch trie nodes through
root common.Hash // Root hash of theaccount trie for metrics
rootPrefetch common.Hash
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can name it clearly? rootPrefetch is a bit confusing. Or you should add some comments to it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about rootParent? It is the root's parent's state root

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

brilliant-lx
brilliant-lx approved these changes Jul 26, 2022
View reviewed changes
Copy link
Collaborator

@brilliant-lx brilliant-lx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@brilliant-lx brilliant-lx changed the title [R4R] Pipecommit enabletrieprefetcher [R4R] Pipecommit enable trie prefetcher Jul 27, 2022
@unclezoro unclezoro merged commit b27ddbb into bnb-chain:develop_July_200M Jul 29, 2022
setunapo pushed a commit to setunapo/bsc that referenced this pull request Jul 29, 2022
@qinglin89 @setunapo
[R4R] Pipecommit enable trie prefetcher (bnb-chain#992)
a2d5f63
@setunapo setunapo mentioned this pull request Jul 29, 2022
Merged
brilliant-lx pushed a commit that referenced this pull request Aug 1, 2022
@qinglin89 @brilliant-lx
[R4R] Pipecommit enable trie prefetcher (#992)
76e3c90
@pythonberg1997 pythonberg1997 mentioned this pull request Aug 31, 2022
Merged
@unclezoro unclezoro mentioned this pull request Aug 31, 2022
Merged
tolunghk added a commit to tolunghk/bsc that referenced this pull request Oct 7, 2022
@tolunghk
Squashed commit of the following:
a41727b 
commit f3fd0f8
Author: zjubfd <[email protected]>
Date:   Fri Oct 7 11:54:15 2022 +0800

    release: prepare for release v1.1.15 (bnb-chain#1112)

commit 5b35b39
Author: zjubfd <[email protected]>
Date:   Fri Oct 7 11:46:38 2022 +0800

    nanofork: add black list account (bnb-chain#1109)

commit 3354924
Merge: 3d3f0a8 781e567
Author: zjubfd <[email protected]>
Date:   Wed Sep 21 18:15:33 2022 +0800

    Merge pull request bnb-chain#1090 from bnb-chain/develop

    prepare for release v1.1.14

commit 781e567
Author: zjubfd <[email protected]>
Date:   Wed Sep 21 15:54:03 2022 +0800

    release: update version and changelogs

commit 8375939
Author: yutianwu <[email protected]>
Date:   Wed Sep 21 15:44:34 2022 +0800

    overwrite HTTPHost in config.toml via cli (bnb-chain#999)

commit 3c7bccb
Author: Leon <[email protected]>
Date:   Mon Sep 19 11:49:31 2022 +0800

    core: fix potential goroutine leak (bnb-chain#1079)

    * core: fix potential goroutine leak

    * core: fix 0 index, and add ut for routineleaking

commit b051dc6
Author: zhaochonghe <[email protected]>
Date:   Thu May 19 15:25:22 2022 +0800

    core: fix the order of address in queue (#24907)

    reverse the order of address in queue

commit 4fb1e9d
Author: Håvard Anda Estensen <[email protected]>
Date:   Tue May 17 13:56:52 2022 +0200

    all: use strings.EqualFold for string comparison (#24890)

commit 57da9c0
Author: s7v7nislands <[email protected]>
Date:   Mon May 16 17:39:07 2022 +0800

    core/asm: use strings.Builder and fix godoc issues (#24861)

commit ef24fa3
Author: Sina Mahmoodi <[email protected]>
Date:   Thu May 12 21:10:08 2022 +0200

    graphql: fix long literal passed in a variable (#24864)

commit 461835a
Author: s7v7nislands <[email protected]>
Date:   Wed May 11 13:03:35 2022 +0800

    core/vm: clean up some dead functions (#24851)

commit 4feeaf3
Author: aaronbuchwald <[email protected]>
Date:   Tue May 10 10:26:48 2022 -0400

    core/vm: update benchmark to use Errorf instead of Sprintf (#24845)

commit 5d980a0
Author: Sina Mahmoodi <[email protected]>
Date:   Tue May 10 10:46:26 2022 +0200

    internal: drop TestSignCliqueBlock (#24837)

commit e8b9338
Author: Marius van der Wijden <[email protected]>
Date:   Fri May 6 17:20:41 2022 +0200

    core/state/snapshot: fix race condition (#24685)

    Fixes three race conditions found through fuzzing by David Theodore

commit 5a06ca3
Author: Rachel Franks <[email protected]>
Date:   Fri May 6 09:45:57 2022 -0500

    abi: fix checks when all fields are indexed (#24792)

    This PR fixes abi checks in the edge case where all arguments are indexed

commit 8d19185
Author: Martin Holst Swende <[email protected]>
Date:   Fri May 6 15:23:43 2022 +0200

    eth/fetcher: avoid hang in tests (partial fix for #23331) (#23351)

    * eth/fetcher: fix test to avoid hanging. Partial fix for #23331

    * eth/filters: avoid dangling goroutines

    * eth/fetcher: revert closing of proceed

commit f781ea8
Author: ImanSharaf <[email protected]>
Date:   Thu May 5 10:44:36 2022 -0700

    p2p/simulations: escape mockerType value from request (#24822)

    Co-authored-by: Felix Lange <[email protected]>

commit 0981812
Author: EXEC <[email protected]>
Date:   Thu May 5 23:58:43 2022 +0800

    eth/filters: remove explicit continue label in filterLogs (#24795)

    The loop label can be removed because this 'continue' statement
    is not in a nested loop.

commit bd22292
Author: Evgeny Kolyakov <[email protected]>
Date:   Wed May 4 00:37:37 2022 +0300

    build: fix formatted logs (#24807)

    Changed `log.Fatal` to `log.Fatalf()` as it has a parameter...

commit cea8319
Author: Martin Holst Swende <[email protected]>
Date:   Tue May 3 12:12:40 2022 +0200

    cmd/utils: double limit on free-disk monitor (#24781)

    This PR doubles the limit on which to trigger automatic shutdown, and also changes the timer to run once every 30s instead of 60s.

commit 58f1e8b
Author: EXEC <[email protected]>
Date:   Tue May 3 17:17:24 2022 +0800

    docker: speed up docker image build (#24796)

    This PR improves the docker build speed for repeated builds where go.mod and go.sum do no change, by placing the downloaded dependencies in a lower layer

commit f67e303
Author: hero5512 <[email protected]>
Date:   Fri Apr 29 16:07:52 2022 +0800

    eth/filters: remove unused struct fields (#24782)

commit a181eee
Author: Joshua Gutow <[email protected]>
Date:   Wed Apr 27 02:51:24 2022 -0700

    ethclient/gethclient: return storage proofs in GetProof (#24697)

    Storage proofs were being unmarshalled from the RPC form to the go struct, but were not being included in the final returned struct.

commit 34420d0
Author: tia-99 <[email protected]>
Date:   Wed Apr 27 16:56:50 2022 +0800

    core/types: fix unhandled errors in TestTransactionCoding (#24692)

commit ae31b5f
Author: henopied <[email protected]>
Date:   Mon Apr 25 04:05:47 2022 -0500

    p2p: fix type of DiscSubprotocolError (#24747)

    It was 'int' accidentally, should be DiscReason instead.

commit 6aae1bc
Author: Nikita Kozhemyakin <[email protected]>
Date:   Mon Apr 25 10:35:04 2022 +0300

    mobile: fix receipt encoding to json (#24701)

commit 30c695c
Author: jwasinger <[email protected]>
Date:   Sun Apr 24 21:16:49 2022 -1000

    cmd/evm: ensure input length is even (#24721)

    * cmd/evm: ensure input length is even

    * cmd/evm: minor nit + lintfix

    Co-authored-by: Martin Holst Swende <[email protected]>

commit 3766168
Author: Emmanuel T Odeke <[email protected]>
Date:   Fri Apr 22 00:30:15 2022 +0100

    eth/tracers/logger: remove unnecessary comparisons in accessList.equal (#24663)

    This change removes extraneous/unnecessary checks for equality
    when comparing 2 accessList values A and B. Given that we validate that
    their lengths of A and B are equal, if so and if every element in A is
    in B, reflexively every element in B is already in A. If that weren't
    the case and an element g existed in A but not in B, that would mean
    that there is an extra element and hence a mathematical contradiction.

    Fixes #24658

commit 3081421
Author: Darioush Jalali <[email protected]>
Date:   Wed Apr 20 09:12:06 2022 -0500

    trie: remove unused makeHashNode (#24702)

commit a723243
Author: rjl493456442 <[email protected]>
Date:   Tue Apr 12 16:34:07 2022 +0800

    core: fix benchmark panic (#24657)

    This PR fixes a few panics in the chain marker benchmarks. The root
    cause for panic is in chain marker the genesis header/block is not
    accessible, while it's expected to be obtained in tests. So this PR
    avoids touching genesis header at all to avoid panic.

commit f37f2a9
Author: Felix Lange <[email protected]>
Date:   Mon Apr 11 20:19:13 2022 +0200

    core/types: make "miner" optional in Header JSON (#24666)

    "miner" is not set for pending block responses in some cases.

    Fixes #24632

commit a41619f
Author: rjl493456442 <[email protected]>
Date:   Fri Apr 8 21:48:52 2022 +0800

    les: fix panic in ultralight client sync (#24641)

commit f310921
Author: Péter Szilágyi <[email protected]>
Date:   Thu Apr 7 09:36:44 2022 +0300

    core/state/snapshot: remove noop map item assignment

commit 211ab45
Author: Péter Szilágyi <[email protected]>
Date:   Wed Apr 6 10:18:57 2022 +0300

    eth/fetcher: if peers never respond, drop them

commit c54fd7f
Author: Nic Jansma <[email protected]>
Date:   Wed Mar 30 04:28:27 2022 -0400

    eth: change snapshot extension registration failure to warning instead of error (#24475)

    * core: Change Snapshot extension registration failed to Debug

    * Update eth/handler.go

    Co-authored-by: Martin Holst Swende <[email protected]>

commit 3b4c683
Author: Martin Holst Swende <[email protected]>
Date:   Tue Mar 29 22:36:55 2022 +0200

    eth/tracers/logger: use omitempty to reduce log bloat (#24547)

    Makes the evm json output less verbose: omitting output of `memory` and `returndata` in case they are empty.

commit 3d5896a
Author: ucwong <[email protected]>
Date:   Wed Mar 30 04:33:19 2022 +0800

    go.mod : upnp 1.0.3 stable version (#24573)

commit 3d3f0a8
Author: zjubfd <[email protected]>
Date:   Wed Sep 14 14:49:07 2022 +0800

    ci: add curl in docker file

commit b8020b9
Author: zjubfd <[email protected]>
Date:   Tue Sep 6 16:42:21 2022 +0800

    fix:  remove fixed curl version in docker file

commit 6f6bbcf
Merge: 723863e aa835e0
Author: zjubfd <[email protected]>
Date:   Fri Sep 2 11:31:36 2022 +0800

    Merge pull request bnb-chain#1056 from bnb-chain/develop

    prepare for release v1.1.13

commit aa835e0
Author: Roshan <[email protected]>
Date:   Wed Aug 31 17:04:48 2022 +0800

    upgrade: update version and CHANGELOG.md (bnb-chain#1073)

commit fd6fa7c
Author: Roshan <[email protected]>
Date:   Wed Aug 31 14:42:23 2022 +0800

    fix: update cross chain logic of native stake (bnb-chain#1066)

commit 755df16
Merge: 64549a7 24da9ec
Author: zjubfd <[email protected]>
Date:   Wed Aug 31 13:30:59 2022 +0800

    Merge pull request bnb-chain#1068 from qinglin89/dev-rlp

commit 64549a7
Author: Leon <[email protected]>
Date:   Wed Aug 31 13:30:25 2022 +0800

    statedb: get rid of activestate, stopPrefetcher manually (bnb-chain#1006)

commit b485651
Author: bittrainee <[email protected]>
Date:   Wed Aug 31 12:38:09 2022 +0800

    fix: fix potential goroutine leak (bnb-chain#1067)

commit f457598
Author: dylanhuang <[email protected]>
Date:   Wed Aug 31 12:37:20 2022 +0800

    txpool: reheap the priced list if london fork not enabled (bnb-chain#1070)

commit 24da9ec
Author: qinglin89 <[email protected]>
Date:   Fri Aug 26 14:38:01 2022 +0800

    fix: gomod add packages

commit da57d11
Author: Qian Bin <[email protected]>
Date:   Wed Mar 9 21:45:17 2022 +0800

    rlp trie: faster trie node encoding (#24126)

    This change speeds up trie hashing and all other activities that require
    RLP encoding of trie nodes by approximately 20%. The speedup is achieved by
    avoiding reflection overhead during node encoding.

    The interface type trie.node now contains a method 'encode' that works with
    rlp.EncoderBuffer. Management of EncoderBuffers is left to calling code.
    trie.hasher, which is pooled to avoid allocations, now maintains an
    EncoderBuffer. This means memory resources related to trie node encoding
    are tied to the hasher pool.

    Co-authored-by: Felix Lange <[email protected]>

commit 25337f5
Author: Felix Lange <[email protected]>
Date:   Fri Feb 18 08:10:26 2022 +0100

    core/types: faster RLP encoding of Header, StateAcccount, Re... (#24420)

    This change makes use of the new code generator rlp/rlpgen to improve the
    performance of RLP encoding for Header and StateAccount. It also speeds up
    encoding of ReceiptForStorage using the new rlp.EncoderBuffer API.

    The change is much less transparent than I wanted it to be, because Header and
    StateAccount now have an EncodeRLP method defined with pointer receiver. It
    used to be possible to encode non-pointer values of these types, but the new
    method prevents that and attempting to encode unadressable values (even if
    part of another value) will return an error. The error can be surprising and may
    pop up in places that previously didn't expect any errors.

    To make things work, I also needed to update all code paths (mostly in unit tests)
    that lead to encoding of non-pointer values, and pass a pointer instead.

    Benchmark results:

        name                             old time/op    new time/op    delta
        EncodeRLP/legacy-header-8           328ns ± 0%     237ns ± 1%   -27.63%  (p=0.000 n=8+8)
        EncodeRLP/london-header-8           353ns ± 0%     247ns ± 1%   -30.06%  (p=0.000 n=8+8)
        EncodeRLP/receipt-for-storage-8     237ns ± 0%     123ns ± 0%   -47.86%  (p=0.000 n=8+7)
        EncodeRLP/receipt-full-8            297ns ± 0%     301ns ± 1%    +1.39%  (p=0.000 n=8+8)

        name                             old speed      new speed      delta
        EncodeRLP/legacy-header-8        1.66GB/s ± 0%  2.29GB/s ± 1%   +38.19%  (p=0.000 n=8+8)
        EncodeRLP/london-header-8        1.55GB/s ± 0%  2.22GB/s ± 1%   +42.99%  (p=0.000 n=8+8)
        EncodeRLP/receipt-for-storage-8  38.0MB/s ± 0%  64.8MB/s ± 0%   +70.48%  (p=0.000 n=8+7)
        EncodeRLP/receipt-full-8          910MB/s ± 0%   897MB/s ± 1%    -1.37%  (p=0.000 n=8+8)

        name                             old alloc/op   new alloc/op   delta
        EncodeRLP/legacy-header-8           0.00B          0.00B           ~     (all equal)
        EncodeRLP/london-header-8           0.00B          0.00B           ~     (all equal)
        EncodeRLP/receipt-for-storage-8     64.0B ± 0%      0.0B       -100.00%  (p=0.000 n=8+8)
        EncodeRLP/receipt-full-8             320B ± 0%      320B ± 0%      ~     (all equal)

commit 1d6c01a
Author: Felix Lange <[email protected]>
Date:   Wed Feb 16 18:14:12 2022 +0100

    rlp/rlpgen: RLP encoder code generator (#24251)

    This change adds a code generator tool for creating EncodeRLP method
    implementations. The generated methods will behave identically to the
    reflect-based encoder, but run faster because there is no reflection overhead.

    Package rlp now provides the EncoderBuffer type for incremental encoding. This
    is used by generated code, but the new methods can also be useful for
    hand-written encoders.

    There is also experimental support for generating DecodeRLP, and some new
    methods have been added to the existing Stream type to support this. Creating
    decoders with rlpgen is not recommended at this time because the generated
    methods create very poor error reporting.

    More detail about package rlp changes:

    * rlp: externalize struct field processing / validation

    This adds a new package, rlp/internal/rlpstruct, in preparation for the
    RLP encoder generator.

    I think the struct field rules are subtle enough to warrant extracting
    this into their own package, even though it means that a bunch of
    adapter code is needed for converting to/from rlpstruct.Type.

    * rlp: add more decoder methods (for rlpgen)

    This adds new methods on rlp.Stream:

    - Uint64, Uint32, Uint16, Uint8, BigInt
    - ReadBytes for decoding into []byte
    - MoreDataInList - useful for optional list elements

    * rlp: expose encoder buffer (for rlpgen)

    This exposes the internal encoder buffer type for use in EncodeRLP
    implementations.

    The new EncoderBuffer type is a sort-of 'opaque handle' for a pointer to
    encBuffer. It is implemented this way to ensure the global encBuffer pool
    is handled correctly.

commit f2b1f3f
Merge: 2e22fe3 94885ae
Author: zjubfd <[email protected]>
Date:   Tue Aug 23 10:43:14 2022 +0800

    Merge pull request bnb-chain#1061 from qinglin89/dev-merge

    fix: upstream patches

commit 94885ae
Author: Zhang Zhuo <[email protected]>
Date:   Tue Mar 29 16:21:11 2022 +0800

    core/vm: fix sstore gas comment type (#24583)

commit 9c72720
Author: rjl493456442 <[email protected]>
Date:   Wed Mar 23 01:58:05 2022 +0800

    eth/downloader: ignore zero size header batch for importing (#24569)

    * eth/downloader: ignore zero size header batch for importing

    * core, light: reject empty header batch for importing

commit c1d9690
Author: rjl493456442 <[email protected]>
Date:   Wed Mar 16 18:13:10 2022 +0800

    les, tests: fix vflux fuzzer by removing unnecessary panic (#24537)

commit 3572eb7
Author: Guifel <[email protected]>
Date:   Tue Mar 15 10:27:30 2022 +0100

    eth/tracers: use hex encoding for debug_traceBlock argument (#24517)

commit f8e2104
Author: ligi <[email protected]>
Date:   Tue Mar 15 10:24:45 2022 +0100

    build: fix linter install on windows (#24523)

commit d130c0e
Author: rjl493456442 <[email protected]>
Date:   Tue Mar 15 17:23:37 2022 +0800

    trie: fix two issues in trie iterator (#24539)

    * trie: fix memory leak in trie iterator

    In the trie iterator, live nodes are tracked in a stack while iterating.
    Popped node states should be explictly set to nil in order to get
    garbage-collected.

    * trie: fix empty trie iterator

commit 799f041
Author: s7v7nislands <[email protected]>
Date:   Fri Mar 11 23:24:13 2022 +0800

    rpc: fix defer in test (#24490)

    Co-authored-by: Felix Lange <[email protected]>

commit 0e65495
Author: Martin Holst Swende <[email protected]>
Date:   Thu Mar 10 12:46:48 2022 +0100

    eth/protocols/snap: avoid estimating infinite percentage

commit 9843a36
Author: Zachinquarantine <[email protected]>
Date:   Tue Mar 8 18:42:14 2022 -0500

    mobile: remove deprecated Stop function (#24369)

commit 413a42d
Author: Ceyhun Onur <[email protected]>
Date:   Wed Mar 9 02:39:34 2022 +0300

    core/rawdb: add HasCode, HashTrieNode and use them where possible (#24454)

commit 2e22fe3
Author: dylanhuang <[email protected]>
Date:   Mon Aug 22 11:25:31 2022 +0800

    ci: fix the pattern of commit lint (bnb-chain#1063)

commit c3e3b11
Author: dylanhuang <[email protected]>
Date:   Mon Aug 22 11:24:50 2022 +0800

    consensus: fix the GasLimitBoundDivisor (bnb-chain#1060)

commit 02b31ab
Author: dylanhuang <[email protected]>
Date:   Mon Aug 22 11:24:18 2022 +0800

    test: fix TestOfflineBlockPrune (bnb-chain#1062)

commit e5f4fde
Author: Felföldi Zsolt <[email protected]>
Date:   Tue Mar 1 10:36:42 2022 +0100

    les/vflux/client: fix goroutine leak in testIter (#24449)

commit 62d66e4
Author: Guruprasad Kamath <[email protected]>
Date:   Tue Mar 1 10:36:11 2022 +0100

    appveyor.yml: fetch sub-modules recursively (#24451)

commit 7bb1463
Author: Sina Mahmoodi <[email protected]>
Date:   Tue Feb 22 10:45:11 2022 +0100

    graphql: fix nonce for pending accounts (#24443)

commit c1dd16f
Author: b1ackd0t <[email protected]>
Date:   Tue Feb 15 12:44:22 2022 +0300

    internal/ethapi: fix incorrect type on empty slice (#24372)

    * Fixes #24368

    Signed-off-by: 0x6f736f646f <[email protected]>

    * Update internal/ethapi/api.go

    Co-authored-by: Martin Holst Swende <[email protected]>

commit a1db38a
Author: zhiqiangxu <[email protected]>
Date:   Mon Feb 14 19:05:48 2022 +0800

    p2p: reduce the scope of variable dialPubkey (#24385)

    dialPubkey isn't used anywhere else after dialDest.Load, so it should be safe to
    restrict its scope to the if clause.

commit 788b771
Author: Martin Holst Swende <[email protected]>
Date:   Mon Feb 14 09:22:57 2022 +0100

    core/state: fix read-meters + simplify code (#24304)

commit e64fde1
Author: setunapo <[email protected]>
Date:   Thu Aug 18 15:28:13 2022 +0800

    ci: allow merge pull request

commit 11d2bae
Author: Roshan <[email protected]>
Date:   Thu Aug 18 14:36:45 2022 +0800

    bep153: Native Staking on BSC (bnb-chain#1051)

commit ddbe6c4
Author: setunapo <[email protected]>
Date:   Wed Aug 17 15:19:50 2022 +0800

    ci: no type-case check for scope.

commit dbb047b
Merge: 05d8399 895c077
Author: joeycli <[email protected]>
Date:   Wed Aug 17 14:29:11 2022 +0800

    Merge pull request bnb-chain#1053 from bnb-chain/fix_opendatabase_fail

    state: fix offline tool start failed when start with pruneancient

commit 05d8399
Author: dylanhuang <[email protected]>
Date:   Tue Aug 16 19:01:24 2022 +0800

    ci: add commit message lint (bnb-chain#1044)

commit 895c077
Author: joeycli <[email protected]>
Date:   Tue Aug 16 14:02:45 2022 +0800

    fix bug bnb-chain#1034, offline tool start failed when start with pruneancient

commit 76e3c90
Author: Leon <[email protected]>
Date:   Fri Jul 29 14:21:37 2022 +0800

    [R4R] Pipecommit enable trie prefetcher (bnb-chain#992)

commit 77c8372
Author: setunapo <[email protected]>
Date:   Tue Jul 26 22:23:55 2022 +0800

    Trie prefetch on state pretch (bnb-chain#996)

    * feature: do trie prefetch on state prefetch

    Currently, state prefetch just pre execute the transactions and discard the results.
    It is helpful to increase the snapshot cache hit rate.
    It would be more helpful, if it can do trie prefetch at the same time, since the it will
    preload the trie node and build the trie tree in advance.
    This patch is to implement it, by reusing the main trie prefetch and doing finalize after
    transaction is executed.

    * some code improvements for trie prefetch

    ** increase pendingSize before dispatch tasks
    ** use throwaway StateDB for TriePrefetchInAdvance and remove the prefetcherLock
    ** remove the necessary drain operation in trie prefetch mainloop,
       trie prefetcher won't be used after close.

commit df3e1be
Author: setunapo <[email protected]>
Date:   Fri Jul 15 19:17:08 2022 +0800

    [Feature]: Improve trie prefetch (bnb-chain#952)

    * trie prefetcher for From/To address in advance

    We found that trie prefetch could be not fast enough, especially trie prefetch of
    the outer big state trie tree.
    Instead of do trie prefetch until a transaction is finalized, we could do trie prefetch
    in advance. Try to prefetch the trie node of the From/To accounts, since their root hash
    are most likely to be changed.

    * Parallel TriePrefetch for large trie update.

    Currently, we create a subfetch for each account address to do trie prefetch. If the address
    has very large state change, trie prefetch could be not fast enough, e.g. a contract modified
    lots of KV pair or a large number of account's root hash is changed in a block.

    With this commit, there will be children subfetcher created to do trie prefetch in parallell if
    the parent subfetch's workload exceed the threshold.

    * some improvemnts of parallel trie prefetch implementation

    1.childrenLock is removed, since it is not necessary
      APIs of triePrefetcher is not thread safe, they should be used sequentially.
      A prefetch will be interrupted by trie() or clos(), so we only need mark it as
      interrupted and check before call scheduleParallel to avoid the concurrent access to paraChildren
    2.rename subfetcher.children to subfetcher.paraChildren
    3.use subfetcher.pendingSize to replace totalSize & processedIndex
    4.randomly select the start child to avoid always feed the first one
    5.increase threshold and capacity to avoid create too many child routine

    * fix review comments

    ** nil check refine
    ** create a separate routine for From/To prefetch, avoid blocking the cirtical path

    * remove the interrupt member

    * not create a signer for each transaction

    * some changes to triePrefetcher

    ** remove the abortLoop, move the subfetcher abort operation into mainLoop
       since we want to make subfetcher's create & schedule & abort within a loop to
       avoid concurrent access locks.

    ** no wait subfetcher's term signal in abort()
       it could speed up the close by closing subfetcher concurrently.
       we send stop signnal to all subfetchers in burst and wait their term signal later.

    * some coding improve for subfetcher.scheduleParallel

    * fix a UT crash of s.prefetcher == nil

    * update parallel trie prefetcher configuration

    tested with different combination of parallelTriePrefetchThreshold & parallelTriePrefetchCapacity,
    found the most efficient configure could be:
      parallelTriePrefetchThreshold = 10
      parallelTriePrefetchCapacity  = 20

    * fix review comments: code refine

commit 51bfeca
Author: kyrie-yl <[email protected]>
Date:   Thu Jul 14 17:09:44 2022 +0800

    broadcast block before commit block and add metrics (bnb-chain#975)

    Signed-off-by: cryyl <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@unclezoro unclezoro unclezoro approved these changes

@forcodedancing forcodedancing forcodedancing approved these changes

@brilliant-lx brilliant-lx brilliant-lx approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@qinglin89 @unclezoro @forcodedancing @brilliant-lx