Extend balances table for better Rosetta #9859
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
psteckler
added
the
ci-build-me
bkase
approved these changes
Nov 22, 2021
| ); | ||
|
|
||
| CREATE INDEX idx_balances_id ON balances(id); | ||
| CREATE INDEX idx_balances_public_key_id ON balances(public_key_id); | ||
| CREATE INDEX idx_balances_height_seq_nos ON balances(block_height,block_sequence_no,block_secondary_sequence_no); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add extra columns to the
balancestable in the archive db schema, to speed up Rosetta queries and make them less resource-intensive.Update the archive processor to populate those extra columns.
There's a new app,
migrate_balances_tableto convert existing databases to the new format. Tested that app by migrating the replayer test database, then running the replayer, which verifies balances after running each transaction. The replayer found bugs in the migration app. After fixing those bugs, the replayer succeeded.The migration app add cursor tables to the db, so that if it crashes, the app can be restarted and resuming (approximately) where it left off. The app does not remove those tables, they can be DROPped manually (or I can add an automatic DROP if desired). Such restart capability is needed because Postgresql consumes a lot of memory during the migration -- the 32 Gb on my desktop was exhausted about 1/3 the way through the migration.
(The code for the migration app is a bit quick-n-dirty, sorry, but we'll only need to run it once for each archive db.)
Also: updated
extract_blocksto accommodate these changes. After migrating the replayer archive test db, had to run the script to add the chain status column. Ranextract_blockson a small subchain to test.