[R4R]add sharedStorage for prefetching to L1 #792

flywukong · 2022-03-11T08:42:08Z

Description

In the previous prefetch behavior of importing blocks, the prefetch and the main process use completely different state dbs, the prefetch process can only put the retrieved data into the L3 layer, if the main process wants to access, it must go through the L1-L2-L3 layers, the new design let the main process and the prefetch process share a shared_pool with the stateDB. This pool stores the originStorage data (each stateobjects has the corresponding
entry ). By this way , the prefetch data that can be directly fished to the L1 layer
This shared_pool can be represented as the following secondary map structure, which is a private member variable in stateDB , each stateObjects has pointer to access the Corresponding entry of shared_pool

This PR also remove the overhead of TryPreload which cost 2%-3% in main process.

Rationale

we can make a lot of data in the main process that originally need to be accessed in L3 directly accessed in L1 layer, which reduces a lot of L2 and L3 layer overhead in the number of layers

Performance

The two test machine is on AWS , the two nodes has worked in fullsync mode and has caught up to the latest block

This is the delay of process block , we can see the machine which running master branch data has nearly same performance data

Then update a machine with this PR code, it can be clearly seen that the delay data is reduced (green performance data), and the average process delay is reduced by about 7%

The proportion of I/O overhead in the total block cost of the main process dropped from 33% to 30.5%

The overhead of total block cost has reduce from to 3.67% to 1.1% with PR code

Example

add an example CLI or API response...

Changes

Notable changes:

add each change in a bullet point here
...

RenRick

LGTM

RenRick · 2022-03-16T07:05:27Z

core/state/shared_pool.go

+// sharedStorage is used to store maps of originStorage of stateObjects
+type SharedStorage struct {
+ poolLock *sync.RWMutex
+ shared_map map[common.Address]*sync.Map


"shared_map" needs to keep the same naming style with poolLock

setunapo · 2022-03-16T22:45:40Z

core/blockchain.go

 go func(start time.Time, followup *types.Block, throwaway *state.StateDB, interrupt *uint32) {
 bc.prefetcher.Prefetch(followup, throwaway, bc.vmConfig, &followupInterrupt)
- }(time.Now(), block, throwaway, &followupInterrupt)
+ }(time.Now(), block, statedb, &followupInterrupt)


Would not suggest to share StateDB between Prefetcher and main StateDB.
Use prefetch DB instead, main StateDB can access prefetch DB to get the originStorage

Prefetch DB could be used as: L1.5 Cache

The stateDB of main process an prefetcher are different, they only share the sharedpool which store the originStroage maps , I had adjust the sharedPool to be L1.5 Cache by the new commit

setunapo · 2022-03-16T22:46:53Z

core/blockchain.go

 go func(start time.Time, followup *types.Block, throwaway *state.StateDB, interrupt *uint32) {
 bc.prefetcher.Prefetch(followup, throwaway, bc.vmConfig, &followupInterrupt)
- }(time.Now(), block, throwaway, &followupInterrupt)
+ }(time.Now(), block, statedb, &followupInterrupt)


Would not suggest to share StateDB between Prefetcher and main StateDB.
Use prefetch DB instead, main StateDB can access prefetch DB to get the originStorage

setunapo · 2022-03-17T07:48:15Z

core/state/shared_pool.go

+ storage.RUnlock()
+ if !ok {
+ m := new(sync.Map)
+ storage.Lock()


concurrent race condition?

yes, newObjects called by main preoces or prefetcher will both call this function to check or update the sharedpool

core/state/state_object.go

core/state/shared_pool.go

core/state/state_object.go

core/state/statedb.go

unclezoro

Please fix lint.

setunapo · 2022-03-24T08:21:07Z

core/state/shared_pool.go

+ storageMap, ok := s.sharedMap[address]
+ s.RUnlock()
+ if !ok {
+ m := new(sync.Map)


Has race condition.

forcodedancing · 2022-03-25T07:15:19Z

core/state/shared_pool.go

+ }
+}
+
+// Check whether the storage exist in pool,


Would be better to follow the commenting style used in go-ethereum, like, // Copy creates a deep, independent copy of the state. // Snapshots of the copied state cannot be applied to the copy. func (s *StateDB) Copy() *StateDB {

unclezoro · 2022-03-25T07:21:04Z

LGTM

richardrich975

LGTM

core/state/state_object.go

setunapo · 2022-03-24T08:37:30Z

core/state/state_object.go

@@ -204,7 +236,8 @@ func (s *StateObject) GetCommittedState(db Database, key common.Hash) common.Has
 if value, pending := s.pendingStorage[key]; pending {
 return value
 }
- if value, cached := s.originStorage[key]; cached {
+
+ if value, cached := s.getOriginStorage(key); cached {


Can leave it unchanged; the API: getOriginStorage can be removed;
// Try to get it from origin storage prefetch pool
if s.sharedOriginStorage != nil {
val, ok := s.sharedOriginStorage.Load(key)
if ok {
s.originStorage[key] = val.(common.Hash)
return val.(common.Hash), true
}
}

setunapo · 2022-03-24T08:46:21Z

core/state/state_object.go

+ return common.Hash{}, false
+}
+
+func (s *StateObject) setOriginStorage(key common.Hash, value common.Hash) {


No need to add a new APIsetOriginStorage either

setunapo · 2022-03-24T08:56:03Z

core/state/shared_pool.go

@@ -0,0 +1,37 @@
+package state


How about rename the file with: shared_storage_pool.go, shared_pool.go is a bit confusing, shared what?

setunapo · 2022-03-24T09:03:06Z

core/state/statedb.go

@@ -1633,3 +1575,7 @@ func (s *StateDB) GetDirtyAccounts() []common.Address {
 }
 return accounts
 }
+
+func (s *StateDB) GetStorage(address common.Address) *sync.Map {


No need to addGetStorage;
GetStorage is paired with SetStorage, it is for fakeStorage.
Do not occupy the name.

setunapo · 2022-03-24T09:08:41Z

core/state/state_object.go

@@ -79,7 +80,9 @@ type StateObject struct {
 trie Trie // storage trie, which becomes non-nil on first access
 code Code // contract bytecode, which gets set when code is loaded

- originStorage Storage // Storage cache of original entries to dedup rewrites, reset for every transaction
+ sharedOriginStorage *sync.Map // Storage cache of original entries to dedup rewrites, reset for every transaction


// Storage cache of original entries to dedup rewrites, reset for every transaction
the comment is for originStorage, pls add a new comment for sharedOriginStorage

setunapo · 2022-03-24T09:14:23Z

core/state/state_object.go

@@ -120,14 +123,21 @@ func newObject(db *StateDB, address common.Address, data Account) *StateObject {
 if data.Root == (common.Hash{}) {
 data.Root = emptyRoot
 }
+ var storageMap *sync.Map


rename storageMap to sharedStorage, storage is a Map by itself.

setunapo · 2022-03-24T09:16:40Z

core/state/statedb.go

@@ -101,6 +98,8 @@ type StateDB struct {
 stateObjectsPending map[common.Address]struct{} // State objects finalized but not yet written to the trie
 stateObjectsDirty map[common.Address]struct{} // State objects modified in the current execution

+ storagePool *StoragePool // sharedPool to store L1 originStorage of stateObjects


name: storagePool -> originStoragePool to make it clear.

setunapo · 2022-03-24T09:17:40Z

core/state/statedb.go

@@ -101,6 +98,8 @@ type StateDB struct {
 stateObjectsPending map[common.Address]struct{} // State objects finalized but not yet written to the trie
 stateObjectsDirty map[common.Address]struct{} // State objects modified in the current execution

+ storagePool *StoragePool // sharedPool to store L1 originStorage of stateObjects
+ writeOnSharedStorage bool // Write to the shared origin storage of a stateObject while reading from the underlying storage layer.


rename writeOnSharedStorage to updateSharedOriginStorage

* add sharedStorage for prefetching to L1 * remote originStorage in stateObjects * fix core * fix bug of sync map * remove read lock when get & set keys * statedb copy use CopyWithSharedStorage * reduce lock access * fix comment * avoid sharedPool effects on other modules * remove tryPreload * fix comment * fix var name * fix lint * fix L1 miss data && data condition * fix comment

Description: This PR refers to the code optimization done by `BSC`, which mainly includes the following parts: 1. Do BlockBody verification concurrently. 2. Do the calculation of intermediate root concurrently. 3. Commit the MPTs concurrently. 4. Preload accounts before processing blocks. 5. Make the snapshot layers configurable. 6. Reuse some objects to reduce GC. 7. Add shared_pool for `stateDB` to improve cache usage. References: - bnb-chain/bsc#257 - bnb-chain/bsc#792 --------- Co-authored-by: j75689 <[email protected]> Co-authored-by: realowen <[email protected]>

Description This PR refers to the code optimization done by `BSC`, which mainly includes the following parts: 1. Do BlockBody verification concurrently. 2. Do the calculation of intermediate root concurrently. 3. Commit the MPTs concurrently. 4. Preload accounts before processing blocks. 5. Make the snapshot layers configurable. 6. Reuse some objects to reduce GC. 7. Add shared_pool for `stateDB` to improve cache usage. References - bnb-chain/bsc#257 - bnb-chain/bsc#792 --------- Co-authored-by: j75689 <[email protected]> Co-authored-by: realowen <[email protected]>

Description This PR refers to the code optimization done by `BSC`, which mainly includes the following parts: 1. Do BlockBody verification concurrently. 2. Do the calculation of intermediate root concurrently. 3. Commit the MPTs concurrently. 4. Preload accounts before processing blocks. 5. Make the snapshot layers configurable. 6. Reuse some objects to reduce GC. 7. Add shared_pool for `stateDB` to improve cache usage. References - bnb-chain/bsc#257 - bnb-chain/bsc#792 --------- Co-authored-by: j75689 <[email protected]>

flywukong changed the title ~~add sharedStorage for prefetching to L1~~ [WIP]add sharedStorage for prefetching to L1 Mar 11, 2022

add sharedStorage for prefetching to L1

8bccd6f

flywukong force-pushed the sharedStorage branch from 2840a8c to 8bccd6f Compare March 11, 2022 09:06

remote originStorage in stateObjects

6626e02

flywukong force-pushed the sharedStorage branch from 8ecd085 to 3338f74 Compare March 14, 2022 08:20

fix core

7abd888

flywukong force-pushed the sharedStorage branch from 3338f74 to 7abd888 Compare March 14, 2022 08:49

flywukong added 2 commits March 15, 2022 12:26

fix bug of sync map

57f224a

remove read lock when get & set keys

9a9b231

flywukong requested review from j75689, unclezoro and RenRick March 16, 2022 05:15

RenRick reviewed Mar 16, 2022

View reviewed changes

setunapo reviewed Mar 16, 2022

View reviewed changes

statedb copy use CopyWithSharedStorage

55349e0

setunapo reviewed Mar 17, 2022

View reviewed changes

reduce lock access

e031e4e

flywukong force-pushed the sharedStorage branch from f886006 to e031e4e Compare March 18, 2022 04:13

j75689 reviewed Mar 18, 2022

View reviewed changes

core/state/state_object.go Outdated Show resolved Hide resolved

flywukong added 2 commits March 20, 2022 18:01

fix comment

63edb9d

avoid sharedPool effects on other modules

85bd395

flywukong force-pushed the sharedStorage branch from 5c2222a to 85bd395 Compare March 21, 2022 10:16

remove tryPreload

a7a785c