first fruit of debugging tool, block 49018 statediff result

[jangko]

{
  "before": [
    {
      "name": "contract",
      "address": "630ea66c8c5dc205d45a978573fa86df5af1fe7a",                  
      "nonce": "0000000000000000",
      "balance": "0",
      "codeHash": "",
      "storage": {}
    },
    {
      "name": "sender",
      "address": "3d0768da09ce77d25e2d998e6a7b6ed4b9116c2d",
      "nonce": "000000000000000A",
      "balance": "43f0cf1107e922ce",
      "codeHash": "",
      "storage": {}
    },
    {
      "name": "miner",
      "address": "bb7b8287f3f0a933474a79eae42cbca977791171",
      "nonce": "0000000000000003",
      "balance": "47d269fc4069450d050",
      "codeHash": "",
      "storage": {}
    }
  ],
  "after": [
    {
      "name": "contract",
      "address": "630ea66c8c5dc205d45a978573fa86df5af1fe7a",
      "nonce": "0000000000000000",
      "balance": "0",
      "codeHash": "",
      "storage": {}  //// HERE IS THE PROBLEM
    },
    {
      "name": "sender",
      "address": "3d0768da09ce77d25e2d998e6a7b6ed4b9116c2d",
      "nonce": "000000000000000B",
      "balance": "43d6743654e7182e",
      "codeHash": "",
      "storage": {}
    },
    {
      "name": "miner",
      "address": "bb7b8287f3f0a933474a79eae42cbca977791171",
      "nonce": "0000000000000003",
      "balance": "47d6c1db0638c46daf0",
      "codeHash": "",
      "storage": {}
    }
  ]
}

//// IT SHOULD BE LIKE THIS
{
  "name": "contract",
  "address": "630ea66c8c5dc205d45a978573fa86df5af1fe7a",
  "nonce": "0000000000000000",
  "balance": "0",
  "codeHash": "",
  "storage": {
    "b10e2d527612073b26eecdfd717e6a320cf44b4afac2b0732d9fcbe2b7fa0cf6": "48656c6c6f20576f726c64210000000000000000000000000000000000000000",
    "0000000000000000000000000000000000000000000000000000000000000001": "000000000000000000000000000000000000000000000000000000000000000c",
    "0000000000000000000000000000000000000000000000000000000000000000": "0000000000000000000000003d0768da09ce77d25e2d998e6a7b6ed4b9116c2d"
  }
}

the storage appear at op code trace, but somehow it vanish after transaction execution.
when I try manually insert storage to that address, the block stateRoot now match expected result, problem confirmed.

that problematic account storageRoot seems not updated, very strange. I could not locate the bug in the code. uncommitted memory layer db?

[zah]

An uncommitted memory layer sounds like a likely culprit indeed. Perhaps there is an inappropriate exception being triggered somewhere before the call/transaction execution is complete.

[jangko]

out of curiosity, I poking around AccountStateDb, then I realize, AccountStateDb were diverged in p2p/chain.nim -> persistBlocks.

one version of statedb located in persistBlocks, the other one was carried around by vmState through it's blockHeader.stateRoot.

block 49018 although not the first block with contract, but it is the first block with contract that modified state trie, it calls sstore opcode.

here is what I already try to merge the diverged statedb:

# in state_db.nim I added this
proc reset*(db: AccountStateDB, root: KeccakHash, pruneTrie: bool) =
  db.trie = initSecureHexaryTrie(HexaryTrie(db.trie).db, root, pruneTrie)

# in vm_state_transactions.nim `applyCreateTransaction` proc
  ....
  vmState.blockHeader.stateRoot = db.rootHash # I added this line
  var c = newBaseComputation(vmState, vmState.blockNumber, msg)

  if execComputation(c):
    db.reset(vmState.blockHeader.stateRoot, vmState.chaindb.pruneTrie) # and this one
    db.addBalance(contractAddress, t.value)
 ....

after that modification, nimbus able to sync to block number 49439, then it stopped again.

this time, the problem located in state_db.nim:

proc setCode*(db: var AccountStateDB, address: EthAddress, code: ByteRange) =
  var account = db.getAccount(address)
  let newCodeHash = keccak256.digest code.toOpenArray
  if newCodeHash != account.codeHash:
    account.codeHash = newCodeHash
    db.accountCodes[newCodeHash] = code
    # XXX: this uses the journaldb in py-evm
    # db.trie.put(account.codeHash.toByteRange_Unnecessary, code)
    db.setAccount(address, account)

proc getCode*(db: AccountStateDB, address: EthAddress): ByteRange =
  db.accountCodes.getOrDefault(db.getCodeHash(address))  

looks like both setCode and getCode is not complete, because block number 49439 contains transaction that sends to a contract.

when getCode called, it return nothing, which is not correct, it should return something and the VM run.

here is code in processTransaction:

    else:
      let code = db.getCode(t.to)
      if code.len == 0:
        # Value transfer
        echo "Transfer ", t.value, " from ", sender, " to ", t.to

        db.addBalance(t.to, t.value)
      else:
        # Contract call
        echo "Contract call"

        debug "Transaction", sender, to = t.to, value = t.value, hasCode = code.len != 0
        let msg = newMessage(t.gasLimit, t.gasPrice, t.to, sender, t.value, t.payload, code.toSeq)
        # TODO: Run the vm 

the TODO looks like need to be implemented if we want to have progress, getCode and setCode also need to be completed.

I will not fix this, I will focus to implement debugging tool, but I think this one is crucial to formulate what is actually needed to implement a good/useful debugging tool. (besides opcode tracing and statediff tracing)

[jangko]

although I've located the bug, my fix is not entirely correct. there should be only one state trie. have more than one state trie lying around only invites bug.

[zah]

I think the solution here is to create a new persistent mapping in the back-end database resolving contract hashes to their corresponding contract codes. You would do this by introducing a new DBKeyKind value in storage_types.nim and function such as contractHashKey that creates a database key. You would write the key-value pair directly to the underlying trie.db backing the AccountStateDB.

The fancy JournalDB is used in py-evm to solve the problem of rolling back blocks that were thrown away after a new alternative chain is selected (due to better fork choice score). We can address this problem later.