Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crashing NAT detectors in k8s on GKE #507

Closed
roderik opened this issue Mar 17, 2020 · 5 comments
Closed

Crashing NAT detectors in k8s on GKE #507

roderik opened this issue Mar 17, 2020 · 5 comments
Labels
bug Something isn't working
Milestone
Chupacabra Sprint 62

Comments

@roderik
Copy link
Contributor

roderik commented Mar 17, 2020

Description

I have a network running on k8s with 1.4.0.
I upgraded the containers to 1.4.1 and the node crashes

2020-03-17 15:03:20.022+00:00 | main | INFO  | Besu | Starting Besu version: besu/v1.4.1/linux-x86_64/oracle_openjdk-java-11                                                                                        
2020-03-17 15:03:20.767+00:00 | main | WARN  | Besu | Permissions are disabled. Cannot enable PERM APIs when not using Permissions.                                                                                 
2020-03-17 15:03:20.851+00:00 | main | INFO  | Besu | Connecting to 1 static nodes.                                                                                                                                 
2020-03-17 15:03:22.835+00:00 | main | INFO  | KeyPairUtil | Loaded key 0xde87a32dfad460814aa616add8bc0fe73f13e19f60c4cf8c30e52fa440d39390bdd6bc5a1af24e7a1ac372599b28fadef173be5bd08b4214a4c6c9c54d3796ce from /sec
2020-03-17 15:03:22.857+00:00 | main | WARN  | Besu | Privacy is disabled. Cannot use EEA/PRIV API methods when not using Privacy.                                                                                  
2020-03-17 15:03:22.891+00:00 | main | INFO  | DatabaseMetadata | Lookup database metadata file in data directory: /data                                                                                            
2020-03-17 15:03:23.009+00:00 | main | INFO  | RocksDBKeyValueStorageFactory | Existing database detected at /data. Version 1                                                                                       
2020-03-17 15:03:29.757+00:00 | main | INFO  | ProtocolScheduleBuilder | Protocol schedule created with milestones: [ConstantinopleFix: 0]                                                                          
2020-03-17 15:03:30.275+00:00 | main | INFO  | RunnerBuilder | Detecting NAT service.                                                                                                                               
    at org.hyperledger.besu.nat.kubernetes.KubernetesNatManager.doStart(KubernetesNatManager.java:64) [besu-nat-1.4.1.jar:1.4.1]                                                                                    
    at org.hyperledger.besu.nat.core.AbstractNatManager.start(AbstractNatManager.java:89) [besu-nat-1.4.1.jar:1.4.1]                                                                                                
    at org.hyperledger.besu.nat.NatService.start(NatService.java:88) [besu-nat-1.4.1.jar:1.4.1]                                                                                                                     
    at org.hyperledger.besu.Runner.start(Runner.java:98) [besu-1.4.1.jar:1.4.1]                                                                                                                                     
    at org.hyperledger.besu.cli.BesuCommand.synchronize(BesuCommand.java:1752) [besu-1.4.1.jar:1.4.1]                                                                                                               
    at org.hyperledger.besu.cli.BesuCommand.startSynchronization(BesuCommand.java:1040) [besu-1.4.1.jar:1.4.1]                                                                                                      
    at org.hyperledger.besu.cli.BesuCommand.run(BesuCommand.java:926) [besu-1.4.1.jar:1.4.1]                                                                                                                        
    at picocli.CommandLine.executeUserObject(CommandLine.java:1769) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at picocli.CommandLine.access$900(CommandLine.java:145) [picocli-4.1.4.jar:4.1.4]                                                                                                                               
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) [picocli-4.1.4.jar:4.1.4]                                                                                                                          
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) [picocli-4.1.4.jar:4.1.4]                                                                                                                          
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]                                                                                            
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:61) [besu-1.4.1.jar:1.4.1]                                                                         
    at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:31) [besu-1.4.1.jar:1.4.1]                                                                         
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]                                                                                            
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:1036) [besu-1.4.1.jar:1.4.1]                                                                                                                     
    at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:911) [besu-1.4.1.jar:1.4.1]                                                                                                                      
    at org.hyperledger.besu.Besu.main(Besu.java:49) [besu-1.4.1.jar:1.4.1]                                                                                                                                          
2020-03-17 15:03:31.528+00:00 | main | INFO  | NetworkRunner | Starting Network.                                                                                                                                    
2020-03-17 15:03:31.704+00:00 | nioEventLoopGroup-2-1 | INFO  | RlpxAgent | P2P RLPx agent started and listening on /0.0.0.0:30303.                                                                                 
2020-03-17 15:03:31.712+00:00 | main | INFO  | PeerDiscoveryAgent | Starting peer discovery agent on host=0.0.0.0, port=30303                                                                                       
2020-03-17 15:03:31.712+00:00 | main | INFO  | NatService | Waiting for up to 60 seconds to detect external IP address...                                                                                           
2020-03-17 15:03:31.712+00:00 | main | WARN  | NatService | Caught exception while trying to query NAT external IP address (ignoring): {}                                                                           
java.lang.NullPointerException: null                                                                                                                                                                                
    at java.util.Objects.requireNonNull(Objects.java:221) ~[?:?]                                                                                                                                                    
    at java.util.Optional.<init>(Optional.java:107) ~[?:?]                                                                                                                                                          
    at java.util.Optional.of(Optional.java:120) ~[?:?]                                                                                                                                                              
    at org.hyperledger.besu.nat.NatService.queryExternalIPAddress(NatService.java:123) [besu-nat-1.4.1.jar:1.4.1]                                                                                                   
    at org.hyperledger.besu.ethereum.p2p.discovery.PeerDiscoveryAgent.start(PeerDiscoveryAgent.java:128) [besu-p2p-1.4.1.jar:1.4.1]                                                                                 
    at org.hyperledger.besu.ethereum.p2p.network.DefaultP2PNetwork.start(DefaultP2PNetwork.java:200) [besu-p2p-1.4.1.jar:1.4.1]                                                                                     
    at org.hyperledger.besu.ethereum.p2p.network.NetworkRunner.start(NetworkRunner.java:80) [besu-p2p-1.4.1.jar:1.4.1]                                                                                              
    at org.hyperledger.besu.Runner.start(Runner.java:99) [besu-1.4.1.jar:1.4.1]                                                                                                                                     
    at org.hyperledger.besu.cli.BesuCommand.synchronize(BesuCommand.java:1752) [besu-1.4.1.jar:1.4.1]                                                                                                               
    at org.hyperledger.besu.cli.BesuCommand.startSynchronization(BesuCommand.java:1040) [besu-1.4.1.jar:1.4.1]                                                                                                      
    at org.hyperledger.besu.cli.BesuCommand.run(BesuCommand.java:926) [besu-1.4.1.jar:1.4.1]                                                                                                                        
    at picocli.CommandLine.executeUserObject(CommandLine.java:1769) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at picocli.CommandLine.access$900(CommandLine.java:145) [picocli-4.1.4.jar:4.1.4]                                                                                                                               
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) [picocli-4.1.4.jar:4.1.4]                                                                                                                          
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) [picocli-4.1.4.jar:4.1.4]                                                                                                                          
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]                                                                                            
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:61) [besu-1.4.1.jar:1.4.1]                                                                         
    at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:31) [besu-1.4.1.jar:1.4.1]                                                                         
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]                                                                                            
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:1036) [besu-1.4.1.jar:1.4.1]                                                                                                                     
    at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:911) [besu-1.4.1.jar:1.4.1]                                                                                                                      
    at org.hyperledger.besu.Besu.main(Besu.java:49) [besu-1.4.1.jar:1.4.1]                                                                                                                                          
2020-03-17 15:03:32.012+00:00 | vert.x-eventloop-thread-1 | INFO  | VertxPeerDiscoveryAgent | Started peer discovery agent successfully, on effective host=0.0.0.0 and port=30303                                   
2020-03-17 15:03:32.019+00:00 | main | INFO  | PeerDiscoveryAgent | P2P peer discovery agent started and listening on /0.0.0.0:30303                                                                                
2020-03-17 15:03:32.182+00:00 | main | INFO  | NatService | Waiting for up to 60 seconds to detect external IP address...                                                                                           
2020-03-17 15:03:32.190+00:00 | main | WARN  | NatService | Caught exception while trying to query NAT external IP address (ignoring): {}     
java.lang.NullPointerException: null                                                                                                                                                                                
    at java.util.Objects.requireNonNull(Objects.java:221) ~[?:?]                                                                                                                                                    
    at java.util.Optional.<init>(Optional.java:107) ~[?:?]                                                                                                                                                          
    at java.util.Optional.of(Optional.java:120) ~[?:?]                                                                                                                                                              
    at org.hyperledger.besu.nat.NatService.queryExternalIPAddress(NatService.java:123) [besu-nat-1.4.1.jar:1.4.1]                                                                                                   
    at org.hyperledger.besu.ethereum.p2p.network.DefaultP2PNetwork.setLocalNode(DefaultP2PNetwork.java:371) [besu-p2p-1.4.1.jar:1.4.1]                                                                              
    at org.hyperledger.besu.ethereum.p2p.network.DefaultP2PNetwork.start(DefaultP2PNetwork.java:216) [besu-p2p-1.4.1.jar:1.4.1]                                                                                     
    at org.hyperledger.besu.ethereum.p2p.network.NetworkRunner.start(NetworkRunner.java:80) [besu-p2p-1.4.1.jar:1.4.1]                                                                                              
    at org.hyperledger.besu.Runner.start(Runner.java:99) [besu-1.4.1.jar:1.4.1]                                                                                                                                     
    at org.hyperledger.besu.cli.BesuCommand.synchronize(BesuCommand.java:1752) [besu-1.4.1.jar:1.4.1]                                                                                                               
    at org.hyperledger.besu.cli.BesuCommand.startSynchronization(BesuCommand.java:1040) [besu-1.4.1.jar:1.4.1]                                                                                                      
    at org.hyperledger.besu.cli.BesuCommand.run(BesuCommand.java:926) [besu-1.4.1.jar:1.4.1]                                                                                                                        
    at picocli.CommandLine.executeUserObject(CommandLine.java:1769) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at picocli.CommandLine.access$900(CommandLine.java:145) [picocli-4.1.4.jar:4.1.4]                                                                                                                               
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) [picocli-4.1.4.jar:4.1.4]                                                                                                                          
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) [picocli-4.1.4.jar:4.1.4]                                                                                                                          
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]                                                                                            
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:61) [besu-1.4.1.jar:1.4.1]                                                                         
    at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:31) [besu-1.4.1.jar:1.4.1]                                                                         
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]                                                                                            
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:1036) [besu-1.4.1.jar:1.4.1]                                                                                                                     
    at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:911) [besu-1.4.1.jar:1.4.1]                                                                                                                      
    at org.hyperledger.besu.Besu.main(Besu.java:49) [besu-1.4.1.jar:1.4.1]                                                                                                                                          
2020-03-17 15:03:32.192+00:00 | main | INFO  | DefaultP2PNetwork | Enode URL enode:https://de87a32dfad460814aa616add8bc0fe73f13e19f60c4cf8c30e52fa440d39390bdd6bc5a1af24e7a1ac372599b28fadef173be5bd08b4214a4c6c9c54d3796c
2020-03-17 15:03:32.210+00:00 | main | INFO  | DefaultSynchronizer | Starting synchronizer.                                                                                                                         
2020-03-17 15:03:32.217+00:00 | main | INFO  | FullSyncDownloader | Start full sync.                                                                                                                                
2020-03-17 15:03:32.229+00:00 | main | INFO  | FullSyncTargetManager | No sync target, wait for peers.                                                                                                              
2020-03-17 15:03:37.314+00:00 | EthScheduler-Timer-0 | INFO  | FullSyncTargetManager | No sync target, wait for peers.                                                                                              
2020-03-17 15:03:41.787+00:00 | pool-10-thread-1 | INFO  | IbftProcessor | Shutting down IBFT event processor                                                                                                       
2020-03-17 15:03:41.797+00:00 | Thread-1 | INFO  | DefaultSynchronizer | Stopping synchronizer                                                                                                                      
2020-03-17 15:03:41.807+00:00 | Thread-1 | INFO  | NetworkRunner | Stopping Network.                                                                                                                                
2020-03-17 15:03:41.792+00:00 | main | INFO  | JsonRpcHttpService | Starting JsonRPC service on 0.0.0.0:8545                                                                                                        
2020-03-17 15:03:41.886+00:00 | Thread-1 | INFO  | EthProtocolManager | Stopping eth Subprotocol.                                                                                                                   
2020-03-17 15:03:41.904+00:00 | Thread-1 | INFO  | EthProtocolManager | eth Subprotocol stopped.                                                                                                                    
2020-03-17 15:03:41.905+00:00 | Thread-1 | INFO  | NetworkRunner | Network stopped.                                                                                                                                 
2020-03-17 15:03:41.908+00:00 | Thread-1 | INFO  | AutoTransactionLogBloomCachingService | Shutting down Auto transaction logs caching service.                                                                     
2020-03-17 15:03:41.923+00:00 | Thread-1 | INFO  | KubernetesNatManager | Stopping kubernetes NAT manager.                                                                                                          
2020-03-17 15:03:42.074+00:00 | vert.x-eventloop-thread-0 | INFO  | JsonRpcHttpService | JsonRPC service started and listening on 0.0.0.0:8545                                                                      
2020-03-17 15:03:42.078+00:00 | main | INFO  | GraphQLHttpService | Starting GraphQL HTTP service on 0.0.0.0:8547                                                                                                   
    at org.hyperledger.besu.Runner.lambda$writeBesuNetworksToFile$14(Runner.java:247) ~[besu-1.4.1.jar:1.4.1]                                                                                                       
    at java.util.Optional.ifPresent(Optional.java:183) [?:?]                                                                                                                                                        
    at org.hyperledger.besu.Runner.writeBesuNetworksToFile(Runner.java:244) [besu-1.4.1.jar:1.4.1]                                                                                                                  
    at org.hyperledger.besu.Runner.start(Runner.java:115) [besu-1.4.1.jar:1.4.1]                                                                                                                                    
    at org.hyperledger.besu.cli.BesuCommand.synchronize(BesuCommand.java:1752) [besu-1.4.1.jar:1.4.1]                                                                                                               
    at org.hyperledger.besu.cli.BesuCommand.startSynchronization(BesuCommand.java:1040) [besu-1.4.1.jar:1.4.1]                                                                                                      
    at org.hyperledger.besu.cli.BesuCommand.run(BesuCommand.java:926) [besu-1.4.1.jar:1.4.1]                                                                                                                        
    at picocli.CommandLine.executeUserObject(CommandLine.java:1769) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at picocli.CommandLine.access$900(CommandLine.java:145) [picocli-4.1.4.jar:4.1.4]                                                                                                                               
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) [picocli-4.1.4.jar:4.1.4]                                                                                                                          
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) [picocli-4.1.4.jar:4.1.4]                                                                                                                          
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]                                                                                            
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:61) [besu-1.4.1.jar:1.4.1]                                                                         
    at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:31) [besu-1.4.1.jar:1.4.1]                                                                         
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]                                                                                            
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:1036) [besu-1.4.1.jar:1.4.1]                                                                                                                     
    at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:911) [besu-1.4.1.jar:1.4.1]                                                                                                                      
    at org.hyperledger.besu.Besu.main(Besu.java:49) [besu-1.4.1.jar:1.4.1]                                                                                                                                          
2020-03-17 15:03:42.222+00:00 | main | INFO  | NatService | Waiting for up to 60 seconds to detect external IP address...                                                                                           
2020-03-17 15:03:42.254+00:00 | main | INFO  | AutoTransactionLogBloomCachingService | Starting auto transaction log bloom caching service.                                                                         
2020-03-17 15:03:42.257+00:00 | main | ERROR | Runner | Startup failed                                                          
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@60807fd9[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@34d9df9f[Wr
    at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) ~[?:?]                                                                                                   
    at org.hyperledger.besu.ethereum.eth.manager.MonitoredExecutors$CountingAbortPolicy.rejectedExecution(MonitoredExecutors.java:165) ~[besu-eth-1.4.1.jar:1.4.1]                                                  
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) ~[?:?]                                                                                                                           
    at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340) ~[?:?]                                                                                                 
    at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562) ~[?:?]                                                                                                       
    at org.hyperledger.besu.ethereum.eth.manager.EthScheduler.scheduleFutureTask(EthScheduler.java:149) ~[besu-eth-1.4.1.jar:1.4.1]                                                                                 
    at org.hyperledger.besu.ethereum.api.query.AutoTransactionLogBloomCachingService.start(AutoTransactionLogBloomCachingService.java:67) ~[besu-api-1.4.1.jar:1.4.1]                                               
    at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]                                                                                                                                                       
    at org.hyperledger.besu.Runner.start(Runner.java:116) [besu-1.4.1.jar:1.4.1]                                                                                                                                    
    at org.hyperledger.besu.cli.BesuCommand.synchronize(BesuCommand.java:1752) [besu-1.4.1.jar:1.4.1]                                                                                                               
    at org.hyperledger.besu.cli.BesuCommand.startSynchronization(BesuCommand.java:1040) [besu-1.4.1.jar:1.4.1]                                                                                                      
    at org.hyperledger.besu.cli.BesuCommand.run(BesuCommand.java:926) [besu-1.4.1.jar:1.4.1]                                                                                                                        
    at picocli.CommandLine.executeUserObject(CommandLine.java:1769) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at picocli.CommandLine.access$900(CommandLine.java:145) [picocli-4.1.4.jar:4.1.4]                                                                                                                               
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) [picocli-4.1.4.jar:4.1.4]                                                                                                                          
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) [picocli-4.1.4.jar:4.1.4]                                                                                                                          
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]                                                                                            
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:61) [besu-1.4.1.jar:1.4.1]                                                                         
    at org.hyperledger.besu.cli.util.ConfigOptionSearchAndRunHandler.handle(ConfigOptionSearchAndRunHandler.java:31) [besu-1.4.1.jar:1.4.1]                                                                         
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:1968) [picocli-4.1.4.jar:4.1.4]                                                                                            
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2322) [picocli-4.1.4.jar:4.1.4]                                                                                                                       
    at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:1036) [besu-1.4.1.jar:1.4.1]                                                                                                                     
    at org.hyperledger.besu.cli.BesuCommand.parse(BesuCommand.java:911) [besu-1.4.1.jar:1.4.1]                                                                                                                      
    at org.hyperledger.besu.Besu.main(Besu.java:49) [besu-1.4.1.jar:1.4.1]                                                                                                                                          
java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@60807fd9[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@34d9df9f[Wr
<STREAM> closed

Versions (Add all that apply)

  • Software version: 1.4.1
  • Java version: Official docker container
  • Docker Version: GKE
@roderik
Copy link
Contributor Author

roderik commented Mar 17, 2020

1.4.0 starts up like this in the same instance:

2020-03-17 15:14:41.234+00:00 | main | INFO  | Besu | Starting Besu version: besu/v1.4.0/linux-x86_64/oracle_openjdk-java-11                                                                                     │
│ 2020-03-17 15:14:41.973+00:00 | main | WARN  | Besu | Permissions are disabled. Cannot enable PERM APIs when not using Permissions.                                                                              │
│ 2020-03-17 15:14:42.045+00:00 | main | INFO  | Besu | Connecting to 1 static nodes.                                                                                                                              │
│ 2020-03-17 15:14:43.680+00:00 | main | INFO  | KeyPairUtil | Loaded key 0xde87a32dfad460814aa616add8bc0fe73f13e19f60c4cf8c30e52fa440d39390bdd6bc5a1af24e7a1ac372599b28fadef173be5bd08b4214a4c6c9c54d3796ce from  │
│ 2020-03-17 15:14:43.688+00:00 | main | WARN  | Besu | Privacy is disabled. Cannot use EEA/PRIV API methods when not using Privacy.                                                                               │
│ 2020-03-17 15:14:43.722+00:00 | main | INFO  | DatabaseMetadata | Lookup database metadata file in data directory: /data                                                                                         │
│ 2020-03-17 15:14:43.816+00:00 | main | INFO  | RocksDBKeyValueStorageFactory | Existing database detected at /data. Version 1                                                                                    │
│ 2020-03-17 15:14:44.797+00:00 | main | INFO  | ProtocolScheduleBuilder | Protocol schedule created with milestones: [ConstantinopleFix: 0]                                                                       │
│ 2020-03-17 15:14:46.038+00:00 | main | INFO  | Runner | Starting Ethereum main loop ...                                                                                                                          │
│ 2020-03-17 15:14:46.039+00:00 | main | INFO  | NatService | No NAT environment detected so no service could be started                                                                                           │
│ 2020-03-17 15:14:46.039+00:00 | main | INFO  | NetworkRunner | Starting Network.                                                                                                                                 │
│ 2020-03-17 15:14:46.180+00:00 | nioEventLoopGroup-2-1 | INFO  | RlpxAgent | P2P RLPx agent started and listening on /0.0.0.0:30303.                                                                              │
│ 2020-03-17 15:14:46.182+00:00 | main | INFO  | PeerDiscoveryAgent | Starting peer discovery agent on host=0.0.0.0, port=30303                                                                                    │
│ 2020-03-17 15:14:46.313+00:00 | vert.x-eventloop-thread-1 | INFO  | VertxPeerDiscoveryAgent | Started peer discovery agent successfully, on effective host=0.0.0.0 and port=30303                                │
│ 2020-03-17 15:14:46.319+00:00 | vert.x-eventloop-thread-1 | INFO  | PeerDiscoveryAgent | P2P peer discovery agent started and listening on /0.0.0.0:30303                                                        │
│ 2020-03-17 15:14:46.388+00:00 | main | INFO  | DefaultP2PNetwork | Enode URL enode:https://de87a32dfad460814aa616add8bc0fe73f13e19f60c4cf8c30e52fa440d39390bdd6bc5a1af24e7a1ac372599b28fadef173be5bd08b4214a4c6c9c54d3 │
│ 2020-03-17 15:14:46.392+00:00 | main | INFO  | DefaultSynchronizer | Starting synchronizer.                                                                                                                      │
│ 2020-03-17 15:14:46.395+00:00 | main | INFO  | FullSyncDownloader | Start full sync.                                                                                                                             │
│ 2020-03-17 15:14:46.397+00:00 | main | INFO  | FullSyncTargetManager | No sync target, wait for peers.   ```

@matkt
Copy link
Contributor

matkt commented Mar 17, 2020

A new feature has been added to automatically detect kubernetes. We will investigate to understand what is happening with GKE.

Can you use the --nat-method = NONE option? This should allow BESU to work as before
and that should resolve your issue
https://besu.hyperledger.org/en/stable/Reference/CLI/CLI-Syntax/#nat-method

@timbeiko timbeiko mentioned this issue Mar 20, 2020
Closed
@timbeiko timbeiko added the bug Something isn't working label Mar 23, 2020
@timbeiko timbeiko added this to the Chupacabra Sprint 61 milestone Mar 23, 2020
@matkt
Copy link
Contributor

matkt commented Apr 8, 2020

Hi @roderik
Did this proposal solve your issue and allow you to start besu correctly?
Thanks

@roderik
Copy link
Contributor Author

roderik commented Apr 8, 2020

Yes, 1.4.2 starts with the --nat-method=NONE option

@matkt matkt mentioned this issue Apr 16, 2020
Merged
@matkt
Copy link
Contributor

matkt commented Apr 20, 2020

The issue will be fixed in the next release (1.4.4). Feel free to reopen the issue if you still have it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
Chupacabra Sprint 62
Development

No branches or pull requests
3 participants
@roderik @timbeiko @matkt