DC/OS Confluent Apache Kafka is an automated service that makes it easy to deploy and manage Apache Kafka on DC/OS. Apache Kafka is a distributed high-throughput publish-subscribe messaging system with strong ordering guarantees. Kafka clusters are highly available, fault tolerant, and very durable. For more information on Confluent Kafka, see its documentation or its white paper, Deploying Confluent Platform on Mesosphere Datacenter OS.
The service comes with a reasonable initial configuration for evaluation use. You can customize the service configuration at initial install, and later update once the service is already running through a configuration rollout process. If you just want to try out the service, you can use the default configuration and be up and running within moments.
Interoperating clients and services can take advantage of DC/OS service discovery features to directly access Confluent Kafka via advertised endpoints, regardless of where the instance is located within a DC/OS Cluster.
You can install multiple Kafka instances on DC/OS and manage them independently. This allows different teams within an organization to have isolated instances of the service.
Note: The service is currently in Beta development. There may be bugs, incomplete features, incorrect documentation, or other discrepancies.
- Multiple instances sharing the same physical systems (requires custom port configuration).
- Vertical (resource) and horizontal (increase broker count) scaling.
- Easy redeployment to new systems upon scheduled or unscheduled outages.
- Consistent DNS addresses regardless of where brokers are located in the cluster.
- Node placement can be customized via Placement Constraints.
Install and Customize
Kafka is available in the Universe and can be installed by using either the web interface or the DC/OS CLI.…Read More
If you are using DC/OS 1.10 and the installed service has a version greater than 2.0.0-x:…Read More
Install DC/OS on your cluster. See the documentation for instructions.…Read More
Through Confluent:…Read More
You can make changes to the service after it has been launched. Configuration management is handled by the scheduler process, which in turn handles deploying DC/OS Confluent Kafka Service itself.…Read More
DC/OS Confluent Kafka does not natively support any backup or restore functionality. For more information on Confluent Kafka backup/restore, please see its documentation…Read More
The Confluent Kafka service will be listed as “Unhealthy” when it detects any underreplicated partitions. This error condition usually indicates a malfunctioning broker. Use the dcos beta-confluent-kafka topic under_replicated_partitions and dcos beta-confluent-kafka topic describe
The “disk” configuration value is denominated in MB. We recommend you set the configuration value log_retention_bytes to a value smaller than the indicated “disk” configuration. See the Configuring section for instructions for customizing these values.…Read More
See Confluent Platform 3.3.1 release notes…Read More
Upgrades and Downgrades
The package supports upgrade and rollback between adjacent versions only. For example, to upgrade from version 2 to version 4, you must first complete an upgrade to version 3, followed by an upgrade to version 4.…Read More