1.3.3.11 Manager Fails with Multiple Broker Listeners

[tcjennings]

Manager fails to parse endpoints for Kafka broker with multiple listeners configured. This worked in 1.3.3.7 (but you could not guarantee which listener Manager should use) so suspect related to new Security Protocol specification in cluster definition.

Example errors from Dockerized Kafka broker deployed as Swarm service:

k.m.m.ActorModel$BrokerIdentity$ - Failed to parse endpoint : INSIDE:https://c4458563815b:9092
pgi_kafka-manager.1.s9oxprfaha4t@ip-172-31-41-88    | java.util.NoSuchElementException: key not found: INSIDE
pgi_kafka-manager.1.s9oxprfaha4t@ip-172-31-41-88    |   at scala.collection.MapLike$class.default(MapLike.scala:228) ~[org.scala-lang.scala-library-2.11.8.jar:na]
pgi_kafka-manager.1.s9oxprfaha4t@ip-172-31-41-88    |   at scala.collection.AbstractMap.default(Map.scala:59) ~[org.scala-lang.scala-library-2.11.8.jar:na]
pgi_kafka-manager.1.s9oxprfaha4t@ip-172-31-41-88    |   at scala.collection.MapLike$class.apply(MapLike.scala:141) ~[org.scala-lang.scala-library-2.11.8.jar:na]
pgi_kafka-manager.1.s9oxprfaha4t@ip-172-31-41-88    |   at scala.collection.AbstractMap.apply(Map.scala:59) ~[org.scala-lang.scala-library-2.11.8.jar:na]
pgi_kafka-manager.1.s9oxprfaha4t@ip-172-31-41-88    |   at kafka.manager.model.SecurityProtocol$.apply(model.scala:410) ~[kafka-manager.kafka-manager-1.3.3.11-sans-externalized.jar:na]
pgi_kafka-manager.1.s9oxprfaha4t@ip-172-31-41-88    |   at kafka.manager.model.ActorModel$BrokerIdentity$$anonfun$6$$anonfun$apply$6$$anonfun$7$$anonfun$apply$7.apply(ActorModel.scala:262) ~[kafka-manager.kafka-manager-1.3.3.11-sans-externalized.jar:na]
pgi_kafka-manager.1.s9oxprfaha4t@ip-172-31-41-88    |   at kafka.manager.model.ActorModel$BrokerIdentity$$anonfun$6$$anonfun$apply$6$$anonfun$7$$anonfun$apply$7.apply(ActorModel.scala:259) ~[kafka-manager.kafka-manager-1.3.3.11-sans-externalized.jar:na]
pgi_kafka-manager.1.s9oxprfaha4t@ip-172-31-41-88    |   at scalaz.Validation$.fromTryCatchNonFatal(Validation.scala:397) ~[org.scalaz.scalaz-core_2.11-7.2.4.jar:na]
pgi_kafka-manager.1.s9oxprfaha4t@ip-172-31-41-88    |   at kafka.manager.model.ActorModel$BrokerIdentity$$anonfun$6$$anonfun$apply$6$$anonfun$7.apply(ActorModel.scala:259) [kafka-manager.kafka-manager-1.3.3.11-sans-externalized.jar:na]
pgi_kafka-manager.1.s9oxprfaha4t@ip-172-31-41-88    |   at kafka.manager.model.ActorModel$BrokerIdentity$$anonfun$6$$anonfun$apply$6$$anonfun$7.apply(ActorModel.scala:258) [kafka-manager.kafka-manager-1.3.3.11-sans-externalized.jar:na]

The ZK node for the broker looks like

{"listener_security_protocol_map":{"INSIDE":"PLAINTEXT","OUTSIDE":"PLAINTEXT"},"endpoints":["INSIDE:https://c4458563815b:9092","OUTSIDE:https://ec2-35-162-25-239.us-west-2.compute.amazonaws.com:9094"],"rack":"us-west-2a","jmx_port":9999,"host":"c4458563815b","timestamp":"1501702861441","port":9092,"version":4}

Note that the names in the listener_security_protocol_map are arbitrarily determined by the Broker's admin. In this case "INSIDE" refers to a listener inside a Swarm network and "OUTSIDE" refers to a listener reachable from outside the swarm. In my case I would prefer that KM use the "INSIDE" listener.

It would be great if KM didn't rely on ZK at all for brokers of appropriate version and would determine a cluster's nodes from a bootstrap broker the same way a new consumer/producer does.

[patelh]

Why can't we just rely on the mapping provided in the listener_security_protocol_map ?

[patelh]

Updated to default to "host" and "port" in the outside json when endpoints cannot be parsed in 1.3.3.13

[solsson]

@tcjennings Is this the same error as in #455? I think it was fixed by #484.

[solsson]

you could not guarantee which listener Manager should use

@tcjennings I'm interested in this observation of yours. It seems to confirm Yolean/kubernetes-kafka#192. Have you investigated this?