Experimental RPC endpoints for generating block witnesses #1977

web3-developer · 2024-01-18T08:15:44Z

This change adds two new custom RPC endpoints which may be used by the Portal Network bridge node to get the changed account state for each new block. It would be impractical to query Nimbus for every account and storage slot for every block so these endpoints allow us to get just the updated state and then feed it into the portal state network. Here are the interfaces for the new endpoints:

proc exp_getWitnessByBlockNumber(blockId: BlockIdentifier, statePostExecution: bool): seq[byte]
proc exp_getProofsByBlockNumber(blockId: BlockIdentifier, statePostExecution: bool): seq[ProofResponse]

The first endpoint returns a block witness which is a binary format which follows the spec here: https://github.com/ethereum/portal-network-specs/blob/01a49a8c9bf08121ecde1b9270a6f2f679cb2568/witness.md.
The second endpoint returns a list of proofs for accounts and storage slots in the same format as the eth_getProof endpoint except it returns a list instead of a single proof.

Both endpoints support returning the state from before or after executing the transactions in the block. Each of these options would be useful in different scenarios. For example, stateless block execution would require getting the block witness data from before execution of the transactions in order to execute the transactions against the witness. For the portal network we will likely want to get the list of proofs from after execution of the block because the bridge will simply be forwarding the proofs into the portal state network and it will want the latest updated state after execution.

The new endpoints are disabled by default and can be enabled by supplying the --rpc-api=exp flag. exp is the new JSON-RPC namespace which has been added for experimental endpoints.

This implementation doesn't yet store the block witnesses. It simply fetches the transactions from the requested block, then re-runs the transactions (without persisting to the db) in order to collect the keys of the updated account and storage state which are used to look up and return the account state from before or after the block execution. We only support returning witnesses/proofs for blocks that have been persisted to disk. I believe this is fine at least for now because the portal network only supports feeding in data from the canonical chain, therefore feeding in block data that may be a from a forks etc probably won't be required. Storage of block witnesses in the database will be coming next perhaps in a separate PR.

web3-developer · 2024-01-18T08:32:47Z

fluffy/network/state/experimental/state_proof_types.nim

- account*: Account
- code* : seq[byte]
- storage*: Table[UInt256, UInt256]
-


Moved into the stateless folder because it likely won't be used by Fluffy and I'm now using these types in Nimbus.

web3-developer · 2024-01-18T08:33:18Z

fluffy/network/state/experimental/state_proof_verification.nim

@@ -67,50 +65,3 @@ func verifyContractBytecode*(
 ok()
 else:
 err("hash of bytecode doesn't match the expected code hash")
-


Moved into the stateless folder because it likely won't be used by Fluffy and I'm now using these procs in Nimbus.

web3-developer · 2024-01-18T08:34:40Z

fluffy/tests/test_state_proof_verification.nim

@@ -214,26 +166,4 @@ suite "State Proof Verification Tests":
 for file in genesisFiles:
 let accounts = getGenesisAlloc("fluffy" / "tests" / "custom_genesis" / file)
 var state = accounts.toState()
- checkInvalidProofsWithBadValue(accounts, state[0], state[1])


Tests moved into stateless folder.

web3-developer · 2024-01-18T08:37:55Z

nimbus/core/executor/process_block.nim

+ # trie keep intact, rather than destroyed by trie pruning. But the current
+ # block will still get a pruned trie. If trie pruning deactivated,
+ # `applyDeletes` have no effects.
+ dbTx.commit(applyDeletes = false)


I'm calling processBlock from the new RPC endpoints with commit = false.

web3-developer · 2024-01-18T08:40:45Z

nimbus/evm/state.nim

- {.raises: [CatchableError].} =
+proc buildWitness*(
+ vmState: BaseVMState,
+ mkeys: MultikeysRef): seq[byte] {.raises: [CatchableError].} =


Added this new proc which supports getting a blockWitness by passing in a MultikeysRef. This was required in order to support returning a block witness from before execution of the transactions. Previously getting a block witness would always return the state of the updated accounts from after executing the block of transactions.

web3-developer · 2024-01-18T08:41:27Z

nimbus/evm/state.nim

+proc buildWitness*(
+ vmState: BaseVMState): seq[byte] {.raises: [CatchableError].} =
+ let mkeys = vmState.stateDB.makeMultiKeys()
+ buildWitness(vmState, mkeys)


This is the existing proc refactored to use the new proc above.

web3-developer · 2024-01-18T08:46:03Z

nimbus/rpc/p2p.nim

@@ -29,6 +29,40 @@ type
 BlockHeader = eth_types.BlockHeader
 Hash256 = eth_types.Hash256



Refactored this code out of the RPC endpoint in order to make testing easier. This code is used by both the eth_getProof and exp_getProofsByBlockNumber RPC endpoints.

web3-developer · 2024-01-18T08:50:49Z

stateless/tree_from_witness.nim

+proc accountNode(t: var TreeBuilder, depth: int): NodeKey {.gcsafe.}
+proc accountStorageLeafNode(t: var TreeBuilder, depth: int): NodeKey {.gcsafe.}
+proc hashNode(t: var TreeBuilder, depth: int, storageMode: bool): NodeKey {.gcsafe.}
+proc treeNode(t: var TreeBuilder, depth: int = 0, storageMode = false): NodeKey {.gcsafe.}


I couldn't get my code to compile without adding these {.gcsafe.} pragmas for some reason. It might be related to the conditional compilation using global variables but I didn't get a chance to confirm. Not sure if this will be an issue or not.

It actually good to add gcsafe pragma. They are useful if you call them in a closure proc.

web3-developer · 2024-01-18T08:52:00Z

stateless/witness_verification.nim

+ storage: storage)
+
+ return accounts
+


This proc is used to verify the witness data against a stateRoot and then return the data contained in the witness in a table.

web3-developer · 2024-01-18T08:54:19Z

tests/test_blockchain_json.nim

@@ -198,6 +199,36 @@ proc blockWitness(vmState: BaseVMState, chainDB: CoreDbRef) =
 if root != rootHash:
 raise newException(ValidationError, "Invalid trie generated from block witness")



Testing the getBlockWitness proc which contains the code to execute the block of transactions. Adding it here because we get full test coverage for all scenarios covered by the test_blockchain_json test.

web3-developer · 2024-01-18T08:59:08Z

nimbus/rpc/experimental.nim

+ else:
+ # Reset state to what it was before executing the block of transactions
+ let initialState = BaseVMState.new(blockHeader, com)
+ (initialState.stateDB.rootHash, initialState.buildWitness(mkeys), flags)


We use the statePostExecution boolean flag to determine whether to return the witness using the initial state from before execution of the transactions or the state from after.

web3-developer · 2024-01-18T08:59:52Z

nimbus/rpc/experimental.nim

+ flags: WitnessFlags): seq[ProofResponse] {.raises: [RlpError].} =
+
+ if witness.len() == 0:
+ return @[]


Blocks with no transactions will generate an empty witness.

jangko · 2024-01-18T09:42:23Z

nimbus/core/executor/process_block.nim

+ # trie keep intact, rather than destroyed by trie pruning. But the current
+ # block will still get a pruned trie. If trie pruning deactivated,
+ # `applyDeletes` have no effects.
+ dbTx.commit(applyDeletes = false)


Database 'beginTransaction' should always paired with 'dispose', 'commit', or 'rollback'. If you don't call them in pair, the db tx will stay in memory.

I think what you want to achive is add another layer of db transaction and then dispose it.

Example:
'''Nim
In your rpc method
Tx=Db.BeginTransaction
Do something
Tx.dispose
'''

Yes good point. Thanks for catching this. I remember seeing that code in the eth_call RPC endpoint. I'll update.

Hey @jangko, actually this processBlock proc has the following lines near the top:

var dbTx = vmState.com.db.beginTransaction() defer: dbTx.dispose()

It looks like dispose() will roll back the transaction already in this proc unless I'm missing something?

@jangko If I add another transaction at a higher layer with a begin and then dispose, will the child transaction still be committed if it has a begin and commit? Or will the parent transaction prevent the child transaction from being committed?

I also noticed that when the transactions are being processed the vmState.stateDB.persist is called on the AccountsCache. Will this be a problem? Or is this just pushing it to the database to be handled by the db transaction at the higher layer?

It looks like dispose() will roll back the transaction already in this proc unless I'm missing something?

It is designed to simplify the api, dispose will only rollback the transaction if there is exception raised and commit is not called. But when the transaction already commited, dispose is a noop. Well, actually it is a bit confusing, but we are too lazy to create a better api. It is a legacy from the beginning of nimbus project.

If I add another transaction at a higher layer with a begin and then dispose, will the child transaction still be committed if it has a begin and commit? Or will the parent transaction prevent the child transaction from being committed?

When a database transaction started, it gives you guarantee, no children transaction will reach the database if the parent call rollback or dispose. The answer is no.
In tracer.nim you may see the process can be intercepted and diverted by a custom database to somewhere else. That is the only possible way to alter the database behavior. But as far as I know, block processing only use regular database transaction.

I also noticed that when the transactions are being processed the vmState.stateDB.persist is called on the AccountsCache. Will this be a problem? Or is this just pushing it to the database to be handled by the db transaction at the higher layer?

It will be handled by db transaction at higher layer. There is no backdoor or bypass mechanism. Nim is good at performing some magic, but we don't use it for state db.

jangko · 2024-01-18T09:49:54Z

tests/rpc/experimental_rpc_client.nim

+ eth_api_types,
+ conversions
+
+createRpcSigs(RpcClient, currentSourcePath.parentDir / "experimental_rpc_calls.nim")


Nim-json-rpc have a new set of create rpc signature without the need to load it from file. Try it out.😁

Okay great, can you direct me to where I can see an example? Is Nimbus updated to use this new version?

It is being used in nim-json-rpc test and in nim-web3: 'eth_api.nim' and 'engine_api.nim'.

jangko · 2024-01-18T09:52:12Z

stateless/tree_from_witness.nim

+proc accountNode(t: var TreeBuilder, depth: int): NodeKey {.gcsafe.}
+proc accountStorageLeafNode(t: var TreeBuilder, depth: int): NodeKey {.gcsafe.}
+proc hashNode(t: var TreeBuilder, depth: int, storageMode: bool): NodeKey {.gcsafe.}
+proc treeNode(t: var TreeBuilder, depth: int = 0, storageMode = false): NodeKey {.gcsafe.}


It actually good to add gcsafe pragma. They are useful if you call them in a closure proc.

jangko · 2024-01-18T09:57:07Z

tests/test_rpc_experimental_json.nim

+
+ let RPC_PORT = 8545
+ var
+ rpcServer = newRpcSocketServer(["127.0.0.1:" & $RPC_PORT])


For testing purpose, better to use port zero and let the OS choose the port for you. The new json-rpc library have server.localAddress to get the active port for your client. Doing so will prevent the next test in queue will not fail because of used port(if they use the same port) when your test crash in the middle of execution.

Thanks. Sure, I'll update.

…al network bridge.

…points.

…pport returning state proofs from before or after block execution.

jangko

LGTM

kdeme · 2024-01-23T08:27:57Z

nimbus/rpc/experimental.nim

+ chainDB = com.db
+ blockHash = chainDB.getBlockHash(blockHeader.blockNumber)
+ blockBody = chainDB.getBlockBody(blockHash)
+ vmState = BaseVMState.new(blockHeader, com)


The creation of this witness by executing the txs in the block would only work with an archival (non-pruning) node, which is what we have as pruning is not implemented (afaik).
However, what would the error path be here in the case that we do have state pruning implemented and the state required for that block is not there?

Good question. I did have a think about this part and basically yes we would need to be running an archive node for this to work. If pruning is enabled (in the future) and you query for an older block which doesn't exist in the db then an RPC error would be returned. It's basically a similar scenario to the existing eth_getBlockByNumber RPC endpoint. I did test this manually by running Nimbus locally and confirmed it returns an informative RPC error.

My point was rather regarding the pruning of state data.
But yes, pruning of block data will/can also become applicable after EIP-4444.

Oh I see what you mean. I haven't tested this but the current implementation will likely just fail when trying to initialize the vm to the state of the parent block if the state doesn't exist. It appears to throw the CatchableError from that line. What would you expect the endpoint to return in this scenario? I think a reasonable assumption is that for this endpoint to work you will either need to run an archive node or only query recent blocks as they get created. Maybe I should try catching the CatchableError and rethrowing it with a message that explains the above assumption.

What would you expect the endpoint to return in this scenario?

I would expect a CatchableError, or perhaps something more specific. But I was wondering if we perhaps wouldn't hit a Defect.

I think a reasonable assumption is that for this endpoint to work you will either need to run an archive node or only query recent blocks as they get created.

Yes, I agree. I think when we get pruning, and archive node becomes a cli flag, a quick check can be added in the rpc calls to verify the number the block vs head when the node is not running as archive node. That way, any error within the creation of the state (or usage of partial state) could be avoided.

Ok sure sounds good. I'll check to make sure it doesn't throw a Defect on that line.

It looks like it does throw a defect if the stateRoot doesn't exist on initialization. I'm not sure if it's worth trying to handle this case just yet because it shouldn't be possible to trigger it until we have pruning. Perhaps I should just leave a comment in the code as a reminder to implement the check you suggested above once we have those parameters available?

Yes, a comment sounds good. For now a Defect is fine, as it shouldn't be a case that can occur unless the db is corrupt or so. But when pruning gets added, that additional check may not be forgotten.

kdeme · 2024-01-23T08:31:41Z

nimbus/rpc/experimental.nim

+ com: CommonRef,
+ blockHeader: BlockHeader,
+ statePostExecution: bool): (KeccakHash, BlockWitness, WitnessFlags)
+ {.raises: [CatchableError].} =


We generally prefer to raise more specific errors (and same for catching them). But I assume some (or several) of the calls used in this proc raise directly the CatchableError (possibly due to some forward declaration or callback without raises annotation)?

Good catch, thanks. I believe those exceptions shouldn't occur in practice but I'll update it to use the more specific types in my next PR anyway.

web3-developer requested review from jangko, mjfh and kdeme January 18, 2024 08:15

web3-developer commented Jan 18, 2024

View reviewed changes

jangko requested changes Jan 18, 2024

View reviewed changes

web3-developer added 8 commits January 19, 2024 13:22

Completed draft implementation of witness JSON-RPC endpoints for port…

c84976f

…al network bridge.

Updated Nimbus RPC configuration to support enabling experimental end…

27480ad

…points.

Moved witness verification tests.

5f46765

Added json test for getProof.

bb94c1d

Added main procs to new tests to fix test suite.

090bb24

Added getBlockWitness test to blockchain json test suite.

70d99b1

Added tests for experimental RPC endpoints and improved the API to su…

c104445

…pport returning state proofs from before or after block execution.

Fix broken test.

e9ae629

web3-developer force-pushed the nimbus-witness-custom-rpc-endpoint-1934 branch from 442e9bc to e9ae629 Compare January 19, 2024 05:23

web3-developer mentioned this pull request Jan 19, 2024

Portal State Network - Proof generation & verification #1934

Closed

5 tasks

web3-developer added 2 commits January 19, 2024 16:22

Updates based on PR feedback.

25942f7

Correctly rollback transaction in getBlockWitness proc.

1677cb6

jangko approved these changes Jan 19, 2024

View reviewed changes

web3-developer merged commit 48630cc into master Jan 22, 2024
11 checks passed

web3-developer deleted the nimbus-witness-custom-rpc-endpoint-1934 branch January 22, 2024 09:11

kdeme reviewed Jan 23, 2024

View reviewed changes

		@@ -29,6 +29,40 @@ type
		BlockHeader = eth_types.BlockHeader
		Hash256 = eth_types.Hash256

		@@ -198,6 +199,36 @@ proc blockWitness(vmState: BaseVMState, chainDB: CoreDbRef) =
		if root != rootHash:
		raise newException(ValidationError, "Invalid trie generated from block witness")

Experimental RPC endpoints for generating block witnesses #1977

Experimental RPC endpoints for generating block witnesses #1977

Conversation

web3-developer commented Jan 18, 2024 • edited Loading

Choose a reason for hiding this comment

web3-developer Jan 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

web3-developer Jan 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

web3-developer Jan 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jangko Jan 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jangko left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

web3-developer Jan 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kdeme Jan 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

web3-developer commented Jan 18, 2024 •

edited

Loading

web3-developer Jan 18, 2024 •

edited

Loading

web3-developer Jan 18, 2024 •

edited

Loading

web3-developer Jan 19, 2024 •

edited

Loading

jangko Jan 18, 2024 •

edited

Loading

web3-developer Jan 23, 2024 •

edited

Loading

kdeme Jan 23, 2024 •

edited

Loading