Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot connect to a broker on another machine (LAN, no Docker/Cloud) #1166

Closed
Nervniyak opened this issue Jan 21, 2020 · 5 comments
Closed

Cannot connect to a broker on another machine (LAN, no Docker/Cloud) #1166

Nervniyak opened this issue Jan 21, 2020 · 5 comments
Labels
question

Comments

@Nervniyak
Copy link

Nervniyak commented Jan 21, 2020 •
edited
Loading

Few month i was developing the Kafka application with broker and client on the same machine. Then I gave it to QA and he said that it is not consuming or producing. I tested it out my application with another machine and it didn't work too. Then I tried to test it without my application and without docker just using Confluent (1.0.0) inside console application and Apache Kafka broker (kafka_2.12-2.2.0) on Windows and it failed...

When I try to produce message on client I receive
"No connection could be made because the target machine actively refused it"

And the log file on broker

2020-01-20 16:48:50,101] WARN [SocketServer brokerId=0] Unexpected error from /172.17.0.1; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 104857600)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:104)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:651)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:572)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:483)
	at kafka.network.Processor.poll(SocketServer.scala:830)
	at kafka.network.Processor.run(SocketServer.scala:730)
	at java.lang.Thread.run(Thread.java:748)

However I can fetch metadata successfully
adminClient.GetMetadata(TimeSpan.FromSeconds(10));

producerConfig

            var config = new ProducerConfig
            {
                BootstrapServers = "192.168.1.75:9092",
                MessageTimeoutMs = 2000,
                Debug = "all"
            };

server.properties

num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=0

# SASL with SSL encryption
listeners=PLAINTEXT:https://:9092,SASL_SSL:https://:9093
advertised.listeners=PLAINTEXT:https://127.0.0.1:9092,SASL_SSL:https://127.0.0.1:9093

ssl.client.auth=required

ssl.key.password=fastdata
ssl.keystore.location=certs/kafka.jks
ssl.keystore.password=fastdata
ssl.truststore.location=certs/truststore.jks
ssl.truststore.password=fastdata

ssl.protocol=TLS
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type=JKS
ssl.truststore.type=JKS

# List of enabled mechanisms, can be more than one
sasl.enabled.mechanisms=PLAIN

# Specify one of of the SASL mechanisms
sasl.mechanism.inter.broker.protocol=PLAIN

sasl.jaas.config=config/kafka_server_jaas.conf

#sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";

IP of the broker machine is 192.168.1.75
IP of the client machine is 192.168.1.26

I tried to change advertised.listeners to advertised.listeners=PLAINTEXT:https://192.168.1.75:9092,SASL_SSL:https://127.0.0.1:9093

and now I receive

0-01-21 19:35:57,794] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-48 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2020-01-21 19:35:59,235] WARN [SocketServer brokerId=0] Unexpected error from /192.168.1.26; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369295617 larger than 104857600)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:104)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:640)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:561)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:472)
	at kafka.network.Processor.poll(SocketServer.scala:830)
	at kafka.network.Processor.run(SocketServer.scala:730)
	at java.lang.Thread.run(Thread.java:748)
[2020-01-21 19:36:04,590] WARN [SocketServer brokerId=0] Unexpected error from /192.168.1.26; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369295617 larger than 104857600)
	at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:104)
	at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:424)
	at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:385)
	at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:640)
	at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:561)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:472)
	at kafka.network.Processor.poll(SocketServer.scala:830)
	at kafka.network.Processor.run(SocketServer.scala:730)
	at java.lang.Thread.run(Thread.java:748)

and producer no longer throw with "No connection could be made because the target machine actively refused it", but is stuck in loop

%7|1579627238.901|BRKMAIN|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Enter main broker thread
%7|1579627238.901|WAKEUPFD|rdkafka#producer-1| [thrd:app]: 192.168.1.75:9092/bootstrap: Enabled low-latency ops queue wake-ups
%7|1579627238.901|BROKER|rdkafka#producer-1| [thrd:app]: 192.168.1.75:9092/bootstrap: Added new broker with NodeId -1
%7|1579627238.902|CONNECT|rdkafka#producer-1| [thrd:app]: 192.168.1.75:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
%7|1579627238.902|BRKMAIN|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Enter main broker thread
%7|1579627238.902|INIT|rdkafka#producer-1| [thrd:app]: librdkafka v1.0.0 (0x10000ff) rdkafka#producer-1 initialized (builtin.features gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd, SSL ZLIB SNAPPY SASL_SCRAM PLUGINS HDRHISTOGRAM, debug 0xffff)
%7|1579627238.902|CONNECT|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Received CONNECT op
%7|1579627238.902|STATE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
%7|1579627238.902|BROADCAST|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: Broadcasting state change
%7|1579627238.902|CONNECT|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: broker in state TRY_CONNECT connecting
%7|1579627238.902|STATE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
%7|1579627238.902|BROADCAST|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: Broadcasting state change
%7|1579627238.902|CONNECT|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Connecting to ipv4#192.168.1.75:9092 (plaintext) with socket 712
%7|1579627238.905|CONNECT|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Connected to ipv4#192.168.1.75:9092
%7|1579627238.906|CONNECTED|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Connected (#1)
%7|1579627238.906|FEATURE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
%7|1579627238.906|STATE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
%7|1579627238.906|BROADCAST|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: Broadcasting state change
%7|1579627238.907|SEND|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Sent ApiVersionRequest (v0, 25 bytes @ 0, CorrId 1)
%7|1579627238.916|TOPIC|rdkafka#producer-1| [thrd:app]: New local topic: test-topic
%7|1579627238.916|TOPPARNEW|rdkafka#producer-1| [thrd:app]: NEW test-topic [-1] 00E37FC0 (at rd_kafka_topic_new0:393)
%7|1579627239.900|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic metadata information unknown
%7|1579627239.901|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic partition count is zero: should refresh metadata
%7|1579627239.902|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7|1579627239.903|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
%7|1579627239.903|METADATA|rdkafka#producer-1| [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
%7|1579627239.904|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: no cluster connection
%7|1579627240.905|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic metadata information unknown
%7|1579627240.905|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic partition count is zero: should refresh metadata
%7|1579627240.907|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7|1579627240.908|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
%7|1579627240.908|METADATA|rdkafka#producer-1| [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
%7|1579627240.908|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection
%7|1579627241.909|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic metadata information unknown
%7|1579627241.909|NOINFO|rdkafka#producer-1| [thrd:main]: Topic test-topic partition count is zero: should refresh metadata
%7|1579627241.911|TIMEOUT|rdkafka#producer-1| [thrd:main]: test-topic: 1 message(s) timed out
%7|1579627241.911|CONNECT|rdkafka#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics
%7|1579627241.912|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: refresh unavailable topics
%7|1579627241.912|METADATA|rdkafka#producer-1| [thrd:main]: Skipping metadata refresh of 1 topic(s): no usable brokers
%7|1579627241.912|CONNECT|rdkafka#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 48ms: no cluster connection
Delivery failed: Local: Message timed out
%7|1579627242.015|DESTROY|rdkafka#producer-1| [thrd:app]: Terminating instance (destroy flags none (0x0))
%7|1579627242.015|TERMINATE|rdkafka#producer-1| [thrd:app]: Interrupting timers
%7|1579627242.017|TERMINATE|rdkafka#producer-1| [thrd:app]: Sending TERMINATE to internal main thread
%7|1579627242.018|TERMINATE|rdkafka#producer-1| [thrd:app]: Joining internal main thread
%7|1579627242.018|TERMINATE|rdkafka#producer-1| [thrd:main]: Internal main thread terminating
%7|1579627242.019|DESTROY|rdkafka#producer-1| [thrd:main]: Destroy internal
%7|1579627242.019|BROADCAST|rdkafka#producer-1| [thrd:main]: Broadcasting state change
%7|1579627242.020|DESTROY|rdkafka#producer-1| [thrd:main]: Removing all topics
%7|1579627242.020|TOPPARREMOVE|rdkafka#producer-1| [thrd:main]: Removing toppar test-topic [-1] 00E37FC0
%7|1579627242.021|DESTROY|rdkafka#producer-1| [thrd:main]: test-topic [-1]: 00E37FC0 DESTROY_FINAL
%7|1579627242.021|DESTROY|rdkafka#producer-1| [thrd:main]: Sending TERMINATE to 192.168.1.75:9092/bootstrap
%7|1579627242.021|TERMINATE|rdkafka#producer-1| [thrd:main]: Purging reply queue
%7|1579627242.022|TERMINATE|rdkafka#producer-1| [thrd:main]: Decommissioning internal broker
%7|1579627242.022|TERMINATE|rdkafka#producer-1| [thrd:main]: Join 2 broker thread(s)
%7|1579627242.021|TERM|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Received TERMINATE op in state APIVERSION_QUERY: 2 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 1 waitresps, 0 retrybufs
%7|1579627242.022|TERM|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Received TERMINATE op in state INIT: 1 refcnts, 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs
%7|1579627242.022|BROKERFAIL|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: failed: err: Local: Broker handle destroyed: (errno: No error)
%7|1579627242.023|BROKERFAIL|rdkafka#producer-1| [thrd::0/internal]: :0/internal: failed: err: Local: Broker handle destroyed: (errno: No error)
%7|1579627242.023|FAIL|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Client is terminating (after 3117ms in state APIVERSION_QUERY)
%7|1579627242.023|FAIL|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Client is terminating (after 3123ms in state INIT)
%7|1579627242.024|FEATURE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Updated enabled protocol features -ApiVersion to
%7|1579627242.025|STATE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Broker changed state APIVERSION_QUERY -> DOWN
%7|1579627242.024|STATE|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Broker changed state INIT -> DOWN
%7|1579627242.028|BROADCAST|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: Broadcasting state change
%7|1579627242.028|BROADCAST|rdkafka#producer-1| [thrd::0/internal]: Broadcasting state change
%7|1579627242.028|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Purging bufq with 1 buffers
%7|1579627242.029|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Purging bufq with 0 buffers
%7|1579627242.029|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Purging bufq with 0 buffers
%7|1579627242.030|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Purging bufq with 0 buffers
%7|1579627242.030|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Updating 0 buffers on connection reset
%7|1579627242.030|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Updating 0 buffers on connection reset
%7|1579627242.031|TERMINATE|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Handle is terminating in state DOWN: 1 refcnts (00DDE048), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
%7|1579627242.030|TERMINATE|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Handle is terminating in state DOWN: 1 refcnts (00DE1D28), 0 toppar(s), 0 active toppar(s), 0 outbufs, 0 waitresps, 0 retrybufs: failed 0 request(s) in retry+outbuf
%7|1579627242.031|BROKERFAIL|rdkafka#producer-1| [thrd::0/internal]: :0/internal: failed: err: Local: Broker handle destroyed: (errno: No error)
%7|1579627242.031|BROKERFAIL|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: failed: err: Local: Broker handle destroyed: (errno: No error)
%7|1579627242.032|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Purging bufq with 0 buffers
%7|1579627242.032|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Purging bufq with 0 buffers
%7|1579627242.032|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Purging bufq with 0 buffers
%7|1579627242.033|BUFQ|rdkafka#producer-1| [thrd::0/internal]: :0/internal: Updating 0 buffers on connection reset
%7|1579627242.033|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Purging bufq with 0 buffers
%7|1579627242.035|BUFQ|rdkafka#producer-1| [thrd:192.168.1.75:9092/bootstrap]: 192.168.1.75:9092/bootstrap: Updating 0 buffers on connection reset
%7|1579627242.035|TERMINATE|rdkafka#producer-1| [thrd:main]: Internal main thread termination done
%7|1579627242.036|TERMINATE|rdkafka#producer-1| [thrd:app]: Destroying op queues
%7|1579627242.038|TERMINATE|rdkafka#producer-1| [thrd:app]: Termination done: freeing resources

What am I doing wrong? Why does everything work fine when client and broker are on localhost? Why is metadata fetched without errors, but producing and consuming doesn't connect?
Please help

[ 1.0.0 ] Confluent.Kafka nuget version.
@Nervniyak
Copy link
Author

Update: The error was reproduced on two pairs of machines and with two different brokers: Apache Kafka Broker and Landoop fast-dev

@mhowlett
Copy link
Contributor

this may be relevant: https://rmoff.net/2018/08/02/kafka-listeners-explained/

@OneCricketeer
Copy link

Seems like your messages are too large.

1195725856 larger than 104857600
...
369295617 larger than 104857600

What types of data are you sending?

Maybe relevant (though highly discouraged) - https://stackoverflow.com/questions/21020347/how-can-i-send-large-messages-with-kafka-over-15mb?rq=1

@Nervniyak
Copy link
Author

Nervniyak commented Jan 28, 2020 •
edited
Loading

Thank you for your comments, I read through @mhowlett's link, gained some knowledge on how to set Kafka up, but unfortunately it didn't help me much, because I already had the setup that seemed logically correct.

I had this lines in my server.properties:

listeners=PLAINTEXT:https://192.168.1.75:9092,SASL_SSL:https://:9093
advertised.listeners=PLAINTEXT:https://192.168.1.75:9092,SASL_SSL:https://127.0.0.1:9093

But in the article they used console-producer to test stuff out, so I did try that too, and received different error
ERROR Error when sending message to topic test with key: null, value: 4 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Topic test not present in metadata after 60000 ms.

Running this command
echo "test"|bin\windows\kafka-console-producer.bat --broker-list 192.168.1.75:9092 --topic test-topic
So I googled this new error and found this SO question

Where Arun Y said that you should enter your Host IP in advertised listener, but leave listener empty, so this lines have fixed problem for me (I don't understand why, because the host can reach itself using both localhost and external IP [192.168.1.75])

listeners=PLAINTEXT:https://:9092
advertised.listeners=PLAINTEXT:https://192.168.1.75:9092

@Cricket007 the data I was sending is just "test" message, this error messages about size are definitely wrong.
And the problem has nothing to do with Confluent client, sorry. It is reproducible on kafka-server-start and kafka-console-producer.

@Nervniyak
Copy link
Author

Seems to be related
wurstmeister/kafka-docker#100 (comment)

@Nervniyak Nervniyak changed the title Cannot connect to a broker on another machine Cannot connect to a broker on another machine (LAN, no Docker/Cloud) Jan 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mhowlett @OneCricketeer @Nervniyak