[Node Operator Question] Distributing op-geth in kubernetes cluster

[Padraic-O-Mhuiris]

Are you running the most up to date node software?

• Yes

Did you check the documentation?

• Yes

Did you check for duplicate questions?

• Yes

Issue Description

My team and I have a OP stack deployment using kubernetes and are looking to improve redundancy in case of failover. Ideally we would like to run 2/3 op-geth instances per op-node instance all contained in a cluster and then run 2/3 clusters of the same setup but this is pending the publishing of a stable op-conductor.

As of right now, our issue is that we have a singular operable OP stack cluster using 1 op-node instance and 1 op-geth instance, each within their own stateful set. We have been attempting to scale the number of op-geth instances to 2 but have been having issues getting the additional nodes to sync.

It appears that nodes are able to open a channel of communication. The admin.peers for both nodes identifies the others enode. The geth-node-0, the one which is fully synced and running the chain returns false for eth.syncing as expected but geth-node-1 returns the following output for eth.syncing`

{
  currentBlock: 0,
  healedBytecodeBytes: 0,
  healedBytecodes: 0,
  healedTrienodeBytes: 0,
  healedTrienodes: 0,
  healingBytecode: 0,
  healingTrienodes: 0,
  highestBlock: 0,
  startingBlock: 0,
  syncedAccountBytes: 0,
  syncedAccounts: 0,
  syncedBytecodeBytes: 0,
  syncedBytecodes: 0,
  syncedStorage: 0,
  syncedStorageBytes: 0,
  txIndexFinishedBlocks: 0,
  txIndexRemainingBlocks: 1
}

We have tested if there is a network issue regarding kubernetes but it appears all ports are opened and communication should work. As shown below, the nodiscover is also set as we want to manually control the network peers.

Protocol Description

• op-geth: v1.101315.0
• op-node: v1.7.5
• network: custom
• chainId: custom
• l1chain: holesky & mainnet

Node Logs

geth-node-1

Appears to start as normal but pauses with the Looking for peers being logged occasionally. No signs of attempts to sync with geth-node-0

INFO [05-16|11:09:33.930] Post-Merge hard forks (timestamp based):
INFO [05-16|11:09:33.930]  - Shanghai:                    @0          (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [05-16|11:09:33.930]  - Regolith:                    @0
INFO [05-16|11:09:33.930]  - Canyon:                      @0
INFO [05-16|11:09:33.930]
INFO [05-16|11:09:33.930] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [05-16|11:09:33.930]
INFO [05-16|11:09:33.930] Loaded most recent local block           number=0 hash=257833..54b0bb td=0 age=6d18h27m
INFO [05-16|11:09:33.931] Initialized transaction indexer          range="entire chain"
INFO [05-16|11:09:33.931] Initialising Ethereum protocol           network=7,890,785 dbversion=8
INFO [05-16|11:09:33.931] Loaded local transaction journal         transactions=0 dropped=0
INFO [05-16|11:09:33.931] Chain post-merge, sync via beacon client
INFO [05-16|11:09:33.931] Gasprice oracle is ignoring threshold set threshold=2
WARN [05-16|11:09:33.932] Engine API enabled                       protocol=eth
INFO [05-16|11:09:33.932] Starting peer-to-peer node               instance=Geth/vuntagged-458ce691-20240512/linux-amd64/go1.21.10
INFO [05-16|11:09:33.945] New local node record                    seq=1,715,856,826,044 id=876030cb95507398 ip=127.0.0.1 udp=0 tcp=30303
INFO [05-16|11:09:33.946] Started P2P networking                   self="enode://<ENODE>@127.0.0.1:30303?discport=0"
INFO [05-16|11:09:33.946] IPC endpoint opened                      url=/data/geth/geth.ipc
INFO [05-16|11:09:33.947] Loaded JWT secret file                   path=/jwtsecret crc32=0x34ce5da5
INFO [05-16|11:09:33.947] HTTP server started                      endpoint=[::]:8545 auth=false prefix= cors=* vhosts=*
INFO [05-16|11:09:33.947] WebSocket enabled                        url=ws://[::]:8546
INFO [05-16|11:09:33.947] WebSocket enabled                        url=ws://[::]:8551
INFO [05-16|11:09:33.947] HTTP server started                      endpoint=[::]:8551 auth=true  prefix= cors=localhost vhosts=*
INFO [05-16|11:10:08.945] Looking for peers                        peercount=1 tried=1 static=2

Additional Information

Included below is excerpts of the kubernetes configuration

op-node args:

      containers:
        - name: op-node
          command: ["op-node"]
          args:
            - --l2=http://op-geth-service:8551
            - --l2.jwt-secret=/jwtsecret
            - --sequencer.enabled
            - --sequencer.l1-confs=5
            - --verifier.l1-confs=4
            - --rollup.config=/rollup/rollup.json
            - --rpc.addr=0.0.0.0
            - --rpc.port=8547
            - --p2p.disable
            - --rpc.enable-admin
            - --p2p.sequencer.key=$(SEQUENCER_PRIVATE_KEY)
            - --l1=<L1_URL>
            - --l1.rpckind=debug_geth
            - --metrics.enabled=true

op-geth args:

      containers:
        - name: geth
          args:
            - --datadir=/data/geth
            - --config=/op-geth-config/op-geth-config.toml
            - --nodiscover
            - --http
            - --http.corsdomain=*
            - --http.vhosts=*
            - --http.addr=0.0.0.0
            - --http.api=web3,debug,eth,txpool,net,engine
            - --ws
            - --ws.addr=0.0.0.0
            - --ws.port=8546
            - --ws.origins=*
            - --ws.api=debug,eth,txpool,net,engine
            - --syncmode=full
            - --gcmode=archive
            - --networkid=<L2_CHAINID>
            - --authrpc.vhosts=*
            - --authrpc.addr=0.0.0.0
            - --authrpc.port=8551
            - --authrpc.jwtsecret=/jwtsecret
            - --rollup.sequencerhttp=http://op-node-service:8547
            - --rollup.disabletxpoolgossip=true
            - --metrics
            - --metrics.expensive=true

op-geth config

    [Node.P2P]
    BootstrapNodesV5 = [<GETH_NODE_0_ENR>]
    StaticNodes = [
      "enode://<GETH_NODE_0_ENODE>@<GETH_NODE_0_SERVICE_URL>.svc.cluster.local:30303",
      "enode://<GETH_NODE_1_ENODE>@<GETH_NODE_1_SERVICE_URL>.svc.cluster.local:30303"
    ]
    TrustedNodes = [
      "enode://<GETH_NODE_0_ENODE>@<GETH_NODE_0_SERVICE_URL>.svc.cluster.local:30303",
      "enode://<GETH_NODE_1_ENODE>@<GETH_NODE_1_SERVICE_URL>.svc.cluster.local:30303"
    ]

Feedback

No response

[opfocus]

As of right now, our issue is that we have a singular operable OP stack cluster using 1 op-node instance and 1 op-geth instance, each within their own [stateful set](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/). We have been attempting to scale the number of op-geth instances to 2 but have been having issues getting the additional nodes to sync.
I have limited knowledge about the content you mentioned. The information I can share is my experience of adding replica nodes in the past. If it doesn't match the question you asked, please ignore it, and if you find any mistakes, please let me know.

#17 (comment)

[opfocus]

CC @sbvegan @Chomtana

[Padraic-O-Mhuiris]

@opfocus You're answer is tangential but not directly answering what I'm asking. To clarify, the difficulties we are encountering is in scaling and syncing multiple op-geth instances.

[opfocus]

I don't know if you succeeded,
What I need to add is that I think op-node and op-geth should appear in pairs, because op-geth completely depends on the driver of op-node, and op-geth alone will not work.
But I don't think that affects your purpose. Running two pairs of op-geth + op-node can share the load. But I'm not sure if it helps the reliability of the sequencer.

[Chomtana]

I'm suspecting the op-geth config file may be the cause

--config=/op-geth-config/op-geth-config.toml