KafkaStreamsExecutionContext could expose all properties instead of only the ones under kafka.properties

[SouquieresAdam]

Problem

KStreamplify reads all configurations from the Spring Boot context and exposes those under the kafka.properties prefix in a static global object, KafkaStreamsExecutionContext.

protected void initProperties() { properties = PropertiesUtils.loadProperties(); serverPort = (Integer) properties.get(SERVER_PORT_PROPERTY); kafkaProperties = PropertiesUtils.loadKafkaProperties(properties); KafkaStreamsExecutionContext.registerProperties(kafkaProperties); }

Suggestion

I would like to be able to access all my properties through this object to manage some business or technical aspects that are not directly related to Kafka from my external configuration. Typically, a Kafka Streams application developed with KStreamplify uses Processor or Transformer classes, which are not directly managed by the Spring Application Context.

Alternatives Considered

Possible alternatives for implementation include:

• Hard-coding the kafka.properties prefix that determines what is exposed in the KafkaStreamsExecutionContext.
• Keeping kafka.properties isolated but exposing all properties visible in the Spring Boot configuration context.
• Creating a new application.properties static property category that contains both business and technical parameters we want to expose in the KafkaStreamsExecutionContext static class.
• Using a separate class like ApplicationConfigurationContext, which contains only the non-Kafka business and technical parameters of the Kafka Streams application.

[loicgreffier]

@SouquieresAdam,

Good issue,

I have considered something like:

• Renaming the current properties to kafkaProperties
• In the KafkaStreamsExecutionContext, add a new attribute applicationProperties containing all other properties that are not related to Kafka

But this would break the previous version as users would have to move from getProperties() to getKafkaProperties() and would have to look for properties either on kafkaProperties or applicationProperties.

kafka.properties needs to be kept isolated to:

• Preserve the backward compatibility
• Being easily able to only load the Kafka properties when instantiating the Kafka Streams:

  kstreamplify/kstreamplify-core/src/main/java/com/michelin/kstreamplify/initializer/KafkaStreamsInitializer.java

  Line 92 in 6da5112

  kafkaStreams = new KafkaStreams(topology, KafkaStreamsExecutionContext.getProperties());

I think the option 2 would just offer the simplest approach:

• Just register all properties in the core module:

  kstreamplify/kstreamplify-core/src/main/java/com/michelin/kstreamplify/initializer/KafkaStreamsInitializer.java

  Line 169 in 6da5112

  KafkaStreamsExecutionContext.registerProperties(kafkaProperties);

  and in the Spring Boot module

  kstreamplify/kstreamplify-spring-boot/src/main/java/com/michelin/kstreamplify/initializer/SpringKafkaStreamsInitializer.java

  Line 80 in 6da5112

  KafkaStreamsExecutionContext.registerProperties(kafkaProperties);

• Maybe the KafkaStreamsExecutionContext can expose a getKafkaProperties() in addition to the getProperties() which only return the Kafka properties. Useful to instantiate the Kafka Streams