Confluent Cloud is a fully managed Kafka service that is hosted on Confluent's cloud platform. This saves the user of the ODE the overhead of needing to manage the Kafka and Zookeeper deployments. There are also additional features that Confluent Cloud supports such as schema registry and ksqlDB that could become useful to the ODE in the future. For more information on Confluent Cloud and the additional services they provide, see this link.

Currently, the ODE is able to connect to external Kafka deployments over SASL. This enables the ODE to behave identically but have its data go through Kafka hosted by Confluent Cloud instead (basic/standard/dedicated clusters). Currently, the ODE does not support additional Confluent Cloud features such as Schema Registry and ksqlDB.

The following ODE sub-modules now fully support SASL authentication:

• asn1_codec
• jpo-cvdp
• jpo-sdw-depositor
• jpo-s3-deposit

If you are finding it difficult to keep your Kafka/Zookeeper deployment stable with the data throughput that is required of your use case or project then it might be reasonable to consider Confluent Cloud. It is always good to ensure your Kafka deployments are properly configured first, but sometimes the overhead of managing the Kafka clusters challenging. This can especially be the case when deploying multiple brokers of Kafka in a Kubernetes environment. Offloading the overhead to a fully managed service can help. Confluent Cloud also has developer support.

Deploying the ODE with Confluent Cloud as the targeted Kafka broker endpoint is possible to do once you have your own Confluent Cloud account with at least one Confluent Cloud environment and cluster.

1. curl -L --http1.1 https://cnfl.io/cli | sh -s -- -b /usr/local/bin
2. Verify installation with confluent version

Follow the step-by-step guide on this page: Connect to Confluent. Remember to document the API key you generate and its secret. This is needed for the ODE environment variables.

Confluent Cloud does not allow automatic topic creation so they must be created before the ODE is deployed. This can be done manually through the Confluent Cloud web portal or you can run the CLI command to do this. A shell script can be found in the project that you can run to generate all of the topics, each with XX partitions (8-10 is a good starting point).

The environment variables within your .env file must be properly set to have the ODE and its sub-modules be properly setup to connect to Confluent Cloud. Here is a brief overview of the required variables:

• DOCKER_HOST_IP: must be set to the bootstrap server address (excluding the port). This can be found in your Confluent Cloud's "Cluster Settings" under "Bootstrap server".
• KAFKA_TYPE: must be set to "CONFLUENT". This lets the project know to include the SASL authentication within the consumer and producer connections.
• CONFLUENT_KEY: must be set to the API key being utilized for Confluent Cloud. This is the same key that was generated in step 2.
• CONFLUENT_SECRET: must be set to the API secret being utilized for Confluent Cloud. This is the same secret that was generated in step 2.

Just as the original ODE is deployed, the docker-compose command will be used to do the Confluent Cloud supported ODE deployment. Although there is a separate deployment YAML file for this deployment called docker-compose-confluent-cloud.yml. Run the following commands to perform the deployment:

1. cp docker-compose.yml docker-compose-local.yml
2. cp docker-compose-confluent-cloud.yml docker-compose.yml
3. docker-compose up -d

Confluent Cloud is a helpful service that reduces the overhead of managing your ODE deployment but is not free and you will want to keep an eye on costs. Currently the ODE does not support the schema registry so messages will be sent to Confluent Cloud as plain text in the form of JSON, POJO, or the ASN1_Codec's XML. Confluent Cloud is billed based on data produced, data consumed, and data stored. It is important to perform some preliminary checks with your data throughput with a trial Confluent Cloud account before committing to Confluent Cloud. More information regarding their billing can be found here.