Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[R4R]db: freezer batch compatible offline prunblock command #1005

Merged
merged 5 commits into from
Jul 22, 2022
Merged

[R4R]db: freezer batch compatible offline prunblock command #1005

merged 5 commits into from
Jul 22, 2022

Conversation

joeylichang
Copy link
Contributor

@joeylichang joeylichang commented Jul 20, 2022 •
edited
Loading

Description

  • Freezer batch compatible offline prune tool.
  • Normalized the prune-block command parameters to avoid the first execution failure for new users due to parameter problems.

Changes

Usage

  • geth snapshot prune-block --datadir $datadir_var --datadir.ancient $ancientdir_var
    | --datadir and --datadir.ancient parameters are required options and uses absolute path

Code changes:

  • add offset(after offline prune tool running, offset stands the start block number in ancient) to freezer batch.
  • batch reset and commit will consider offset.

flywukong
flywukong reviewed Jul 20, 2022
View reviewed changes
cmd/geth/snapshot.go Outdated Show resolved Hide resolved
@unclezoro unclezoro changed the title freezer batch compatible offline prunblock command [R4R]db: freezer batch compatible offline prunblock command Jul 20, 2022
Mercybudda
Mercybudda reviewed Jul 20, 2022
View reviewed changes
core/rawdb/database.go Show resolved Hide resolved
core/state/pruner/pruner.go Outdated Show resolved Hide resolved
richardrich975
richardrich975 previously approved these changes Jul 20, 2022
View reviewed changes
Copy link

@richardrich975 richardrich975 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

core/state/pruner/pruner.go Outdated Show resolved Hide resolved
cmd/geth/snapshot.go Outdated Show resolved Hide resolved
Mercybudda
Mercybudda previously approved these changes Jul 20, 2022
View reviewed changes
Copy link
Contributor

@Mercybudda Mercybudda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please pass all UTs check then.

flywukong
flywukong previously approved these changes Jul 20, 2022
View reviewed changes
richardrich975
richardrich975 approved these changes Jul 22, 2022
View reviewed changes
Copy link

@richardrich975 richardrich975 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@unclezoro unclezoro merged commit d93211b into develop Jul 22, 2022
@joeylichang joeylichang deleted the offline_prune_compatible_freezer_batch branch July 23, 2022 15:30
This was referenced Jul 28, 2022
Merged
Merged
jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023
@jsvisa
core/rawdb: freezer batch should implement the offset commit, ref bnb…
1520cb5 
…-chain/bsc#1005

Signed-off-by: Delweng <[email protected]>
jsvisa added a commit to jsvisa/bor that referenced this pull request Feb 21, 2023
@jsvisa
core/rawdb: freezer batch should implement the offset commit, ref bnb…
ecffeda 
…-chain/bsc#1005

Signed-off-by: Delweng <[email protected]>
@jsvisa jsvisa mentioned this pull request Feb 21, 2023
Closed
16 tasks
jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 1, 2023
@jsvisa
core/rawdb: freezer batch should implement the offset commit, ref bnb…
1e2cb93 
…-chain/bsc#1005

Signed-off-by: Delweng <[email protected]>
jsvisa added a commit to jsvisa/bor that referenced this pull request Mar 17, 2023
@jsvisa
core/rawdb: freezer batch should implement the offset commit, ref bnb…
f7833b7 
…-chain/bsc#1005

Signed-off-by: Delweng <[email protected]>
jsvisa added a commit to jsvisa/bor that referenced this pull request Jun 28, 2023
@jsvisa
core/rawdb: freezer batch should implement the offset commit, ref bnb…
8a44857 
…-chain/bsc#1005

Signed-off-by: Delweng <[email protected]>
manav2401 pushed a commit to maticnetwork/bor that referenced this pull request Apr 9, 2024
@jsvisa @manav2401
core/rawdb: freezer batch should implement the offset commit, ref bnb…
cf08b90 
…-chain/bsc#1005

Signed-off-by: Delweng <[email protected]>
manav2401 added a commit to maticnetwork/bor that referenced this pull request May 9, 2024
@manav2401 @jsvisa
Enable ancient block pruning (#1216)
d95c05b 
* core/state: typo

Signed-off-by: Delweng <[email protected]>

* core/rawdb: backport from bnb-chain/bsc#543

Signed-off-by: Delweng <[email protected]>

* eth,ethdb,node,core/state: backport from bnb-chain/bsc#543

Signed-off-by: Delweng <[email protected]>

* eth,core: backport from bnb-chain/bsc#543

Signed-off-by: Delweng <[email protected]>

* cmd: open db with freeze disabled

Signed-off-by: Delweng <[email protected]>

* cli: snapshot prune-block

Signed-off-by: Delweng <[email protected]>

* fix typo

Signed-off-by: Delweng <[email protected]>

* cli/snapshot: fix the issue of dup open db error

Signed-off-by: Delweng <[email protected]>

* cli/snapshot: resolve datadir and ancient before backup

Signed-off-by: Delweng <[email protected]>

* core: more prune-block log

Signed-off-by: Delweng <[email protected]>

* core: truncatetail missing f.offset

Signed-off-by: Delweng <[email protected]>

* core/rawdb: indextx adjust offset of pruned block

Signed-off-by: Delweng <[email protected]>

* core/rawdb: freezer batch should implement the offset commit, ref bnb-chain/bsc#1005

Signed-off-by: Delweng <[email protected]>

* core: check of ancientdb, backport bnb-chain/bsc#817

Signed-off-by: Delweng <[email protected]>

* core/state: read raw borReceipt to backup

Signed-off-by: Delweng <[email protected]>

* core/rawdb: bor receipt maybe in []Receipt or Receipt RLP format

Signed-off-by: Delweng <[email protected]>

* core/state: typo and error msg

Signed-off-by: Delweng <[email protected]>

* core/rawdb: offSet -> offset

Signed-off-by: Delweng <[email protected]>

* cli/snapshot: comment

Signed-off-by: Delweng <[email protected]>

* cli/snapshot: add prune-block doc

Signed-off-by: Delweng <[email protected]>

* docs: add prune-block document

Signed-off-by: Delweng <[email protected]>

* core/rawdb: print wrong bor-receipt length

Signed-off-by: Delweng <[email protected]>

* internal/cli: add snapshot prune block tests (referenced from bsc's PR)

* improve errors

* cmd, core, eth, internal: fix lint

* internal/cli: refactor snapshot prune block test

* fix linters in tests

* internal/cli: add inspect-ancient-db command, update docs

* pruner: use a generic function for simplification

* internal/cli: fixes for inspect-db command

* internal/cli: improve pruning tests

* core/rawdb: update end block calculation logic in inspect command

* core/rawdb: improve checks db initialisation

* core/rawdb: remove offset check

* update mocks for span, ethdb and add command in makefile

* docs/cli: update docs with inspect command

* go mod tidy

* refactor and resolve conflicts

* resolve more conflicts

* refactor

* explicitly read node for hash scheme

* add check for hash scheme, fix tests

* fix typo

* update docs and add warning

* raise error if pbss is enabled

* revert read raw bor receipt change

* consensus/bor: handle nil header case in get root hash

* address comments

* core/rawdb: check chain continuity by matching parent hash

* core/rawdb: account for pruned ancient blocks

* go mod tidy

* fix tests

* fix tests

---------

Signed-off-by: Delweng <[email protected]>
Co-authored-by: Delweng <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@unclezoro unclezoro unclezoro approved these changes

@Mercybudda Mercybudda Mercybudda approved these changes

@calmbeing calmbeing calmbeing approved these changes

@richardrich975 richardrich975 richardrich975 approved these changes

@flywukong flywukong flywukong left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@joeylichang @unclezoro @flywukong @Mercybudda @calmbeing @richardrich975