Deploy a nimbus-eth1 instance connected to the Sepolia network

[zah]

We would like to run one or more nimbus-eth1 nodes as part of our fleet, connected to the upcoming Sepolia network. Since the Sepolia network will transition through the merge soon, each Eth1 node will be paired with a nimbus-eth2 node.

Initially, the pair will run without any validators, but if the setup proves to be stable enough, we might deploy some validators to the consensus node.

Adding consensus layer metadata for the Sepolia network will be tracked here:
status-im/nimbus-eth2#3758

Adding execution layer metadata will be tracked here:
status-im/nimbus-eth1#1130

This issue should track the deployment of a new instance of the https://github.com/status-im/infra-role-beacon-node-linux role connected to a new instance of the https://github.com/status-im/infra-role-nimbus-eth1 role, both configured to connect to the Sepolia network. The two Nimbus roles should respectively use the unstable and the master branch.

[zah]

Sepolia metadata have been added to nimbus-eth2 here:
status-im/nimbus-eth2#3762

[jakubgs]

But there already is a role for deploying nimbus-eth1:
https://github.com/status-im/infra-role-nimbus-eth1

And it's already being used to deploy an nimbus-eth1 node for Rocketpool:

infra-nimbus/ansible/eth1.yml

Line 18 in 5aa3b64

- { role: infra-role-nimbus-eth1, tags: nimbus-eth1 }

Is this somehow different?

[zah]

Ah, right, I had forgotten about this. I've made edits to the issue description.

I'm surprised by your claim that we use the nimbus-eth1 instance for RocketPool. Our production RocketPool node is using a production-ready Eth1 instance (Geth). I think we've decided in the past to just share the same host for the nimbus-eth1 experimental deployment and for our testing RocketPool node. Please correct me if I'm wrong.

[jakubgs]

I think you're right: e7e45739

• Create a new role for nimbus-eth1 and use it on the nimbus.eth1 host. #72

[zah]

Sepolia metadata has been added to nimbus-eth1 in status-im/nimbus-eth1#1148 (the node has to be launched with --network=sepolia, so all prerequisites are in place an we just need to instantiate the nimbus-eth1 role on any server with enough capacity. At the moment syncing the Sepolia network will consume around 9GB of storage.

[jakubgs]

I've deployed both nodes:

[email protected]:/data/beacon-node-sepolia-unstable-trial-01 % s nimbus-eth1-sepolia-master-trial.service | cat
● nimbus-eth1-sepolia-master-trial.service - Nimbus Eth1 node on sepolia network (master)
     Loaded: loaded (/etc/systemd/system/nimbus-eth1-sepolia-master-trial.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2022-07-07 14:15:29 UTC; 9min ago
       Docs: https://github.com/status-im/nimbus-eth1
   Main PID: 3726290 (nimbus)
      Tasks: 20 (limit: 77016)
     Memory: 62.4M
     CGroup: /system.slice/nimbus-eth1-sepolia-master-trial.service
             └─3726290 /data/nimbus-eth1-sepolia-master-trial/repo/build/nimbus --network=sepolia --data-dir=/data/nimbus-eth1-sepolia-master-trial/data/shared_sepolia_0 --nat=extip:65.21.89.157 --log-level=DEBUG --listen-address=0.0.0.0 --tcp-port=30303 --udp-port=30303 --max-peers=160 --discovery=V4 --rpc=true --rpc-address=127.0.0.1 --rpc-port=8567 --ws=true --ws-address=127.0.0.1 --ws-port=9568 --graphql=false --metrics=true --metrics-address=0.0.0.0 --metrics-port=9093

Jul 07 14:15:29 metal-01.he-eu-hel1.nimbus.kiln systemd[1]: Started Nimbus Eth1 node on sepolia network (master).
[email protected]:/data/beacon-node-sepolia-unstable-trial-01 % s beacon-node-sepolia-unstable-trial-01 | cat 
● beacon-node-sepolia-unstable-trial-01.service - Nimbus Beacon Node on sepolia network (unstable)
     Loaded: loaded (/etc/systemd/system/beacon-node-sepolia-unstable-trial-01.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2022-07-07 14:16:58 UTC; 8min ago
       Docs: https://github.com/status-im/nimbus-eth2
    Process: 3728354 ExecStart=/data/beacon-node-sepolia-unstable-trial-01/repo/build/nimbus_beacon_node --network=sepolia --data-dir=/data/beacon-node-sepolia-unstable-trial-01/data/shared_sepolia_0 --web3-url=ws:https://localhost:9568 --log-format=json --log-level=DEBUG --nat=extip:65.21.89.157 --tcp-port=9022 --udp-port=9022 --max-peers=320 --netkey-file=/data/beacon-node-sepolia-unstable-trial-01/data/netkey --slashing-db-kind=v2 --insecure-netkey-password=true --subscribe-all-subnets=false --doppelganger-detection=true --rpc=false --rest=true --rest-address=0.0.0.0 --rest-port=9322 --metrics=true --metrics-address=0.0.0.0 --metrics-port=9222 --validator-monitor-auto --validator-monitor-totals --num-threads=1 (code=exited, status=1/FAILURE)
   Main PID: 3728354 (code=exited, status=1/FAILURE)

Jul 07 14:16:58 metal-01.he-eu-hel1.nimbus.kiln systemd[1]: beacon-node-sepolia-unstable-trial-01.service: Scheduled restart job, restart counter is at 5.
Jul 07 14:16:58 metal-01.he-eu-hel1.nimbus.kiln systemd[1]: Stopped Nimbus Beacon Node on sepolia network (unstable).

Changes:

• status-im/infra-role-nimbus-eth1@527ebe5d - add sepolia to valid network names
• 0b58b2f8 - deploy nimbus-eth1 and beacon nodes for Sepolia

But there is an issue with Beacon node.

[jakubgs]

Looks like it explodes at startup:

{                                                                                          
  "lvl": "FAT",
  "ts": "2022-07-07 14:16:57.831+00:00",
  "msg": "Failed to locate the deposit contract deployment block",
  "topics": "beacnde",
  "depositContract": "0x7f02c3e3c98b133055b8b348b2ac625669ed295d",
  "deploymentBlock": "(isHash: true, hash: 491ebac1b7f9c0eb426047a495dc577140cb3e09036cd3f7266eda86b635d9fa)",
  "err": "Failed to setup web3 connection: WebSocket connection error: 403 Forbidden"
}

Maybe related to JWT?

[jakubgs]

It appears the JWT token is generated at startup if not specified:

[email protected]:/data/beacon-node-sepolia-unstable-trial-01 % /data/nimbus-eth1-sepolia-master-trial/repo/build/nimbus --help | grep -B1 jwt
     --ws-api                  Enable specific set of Websocket RPC API (available: eth, debug) [=Eth].
     --jwt-secret              Path to a file containing a 32 byte hex-encoded shared secret needed for
                               websocket authentication. By default, the secret key is auto-generated.
                               [="jwt.hex" in the data directory (see --data-dir)].

[jakubgs]

Changes necessary to make the API connection secured with JWT tokens work:

• status-im/infra-role-nimbus-eth1@474b200b - add support for specifying JWT shared secret token
• status-im/infra-role-beacon-node-linux@34d59ad6 - add support for specifying shared JWT secret token
• cf57579f - nimbus-sepolia-trial: set JWT shared secret token

And the node is up:

[email protected]:~ % c 0:9322/eth/v1/node/syncing | jq
{
  "data": {
    "head_slot": "123735",
    "sync_distance": "0",
    "is_syncing": false,
    "is_optimistic": true
  }
}

[jakubgs]

The node looks healthy:

[email protected]:~ % /data/nimbus-eth1-sepolia-master-trial/rpc.sh eth_syncing
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "startingBlock": "0x16777a",
    "currentBlock": "0x16837a",
    "highestBlock": "0x16837a"
  }
}
[email protected]:~ % c 0:9093/metrics | grep '^connected_peers '
connected_peers 32.0

And so does the consensus node:

[email protected]:~ % c 0:9322/eth/v1/node/syncing | jq         
{
  "data": {
    "head_slot": "157596",
    "sync_distance": "1",
    "is_syncing": false,
    "is_optimistic": true
  }
}
[email protected]:~ % c 0:9222/metrics | grep '^libp2p_peers '
libp2p_peers 108.0

I consider this done.