DC/OS Confluent Apache Kafka is an automated service that makes it easy to deploy and manage Apache Kafka on DC/OS. Apache Kafka is a distributed high-throughput publish-subscribe messaging system with strong ordering guarantees. Kafka clusters are highly available, fault tolerant, and very durable. For more information on Confluent Kafka, see its documentation or its white paper, Deploying Confluent Platform on Mesosphere Datacenter OS.
The service comes with a reasonable initial configuration for evaluation use. You can customize the service configuration at initial install, and later update once the service is already running through a configuration rollout process. If you just want to try out the service, you can use the default configuration and be up and running within moments.
Interoperating clients and services can take advantage of DC/OS service discovery features to directly access Confluent Kafka via advertised endpoints, regardless of where the instance is located within a DC/OS Cluster.
You can install multiple Kafka instances on DC/OS and manage them independently. This allows different teams within an organization to have isolated instances of the service.
Note: The service is currently in Beta development. There may be bugs, incomplete features, incorrect documentation, or other discrepancies.
- Multiple instances sharing the same physical systems (requires custom port configuration).
- Vertical (resource) and horizontal (increase broker count) scaling.
- Easy redeployment to new systems upon scheduled or unscheduled outages.
- Consistent DNS addresses regardless of where brokers are located in the cluster.
- Node placement can be customized via Placement Constraints.
Install and Customize
Kafka is available in the Universe and can be installed by using either the web interface or the DC/OS CLI.…Read More
In order to run Apache Kafka with Kerberos security enabled, a principal needs to be added for every broker in the cluster. For example, a three node cluster with the default service primary (service.security.kerberos.primary) of kafka will require to following principals:…Read More
If you are using DC/OS 1.10 and the installed service has a version greater than 2.0.0-x:…Read More
Install a Kafka cluster. If you are using open source DC/OS, install a Kafka cluster with the following command from the DC/OS CLI. If you are using Enterprise DC/OS, you may need to follow additional instructions. See the Install and Customize section for more information.…Read More
Through Confluent:…Read More
You can make changes to the service after it has been launched. Configuration management is handled by the scheduler process, which in turn handles deploying DC/OS Kafka Service itself.…Read More
DC/OS Confluent Kafka does not natively support any backup or restore functionality. For more information on Confluent Kafka backup/restore, please see its documentation…Read More
The Kafka service will be listed as “Unhealthy” when it detects any underreplicated partitions. This error condition usually indicates a malfunctioning broker. Use the $ dcos confluent-kafka topic under_replicated_partitions and $ dcos confluent-kafka topic describe
The “disk” configuration value is denominated in MB. We recommend you set the configuration value log_retention_bytes to a value smaller than the indicated “disk” configuration. See the Configuring section for instructions for customizing these values.…Read More
This is a beta release of the DC/OS Confluent Kafka service. It contains multiple improvements as well as new features that are to be considered of beta quality. Do not operate this version in production.…Read More
We support upgrade/rollback between adjacent versions only. Concretely, to upgrade from version 2 to version 4, you must upgrade from 2 -> 3, then from 3 -> 4.…Read More