DC/OS Apache Kafka is an automated service that makes it easy to deploy and manage Apache Kafka on Mesosphere DC/OS, eliminating nearly all of the complexity traditionally associated with managing a Kafka cluster. Apache Kafka is a distributed high-throughput publish-subscribe messaging system with strong ordering guarantees. Kafka clusters are highly available, fault tolerant, and very durable. For more information on Apache Kafka, see the Apache Kafka documentation. DC/OS Kafka gives you direct access to the Kafka API so that existing producers and consumers can interoperate. You can configure and install DC/OS Kafka in moments. Multiple Kafka clusters can be installed on DC/OS and managed independently, so you can offer Kafka as a managed service to your organization.

Benefits

DC/OS Kafka offers the following benefits of a semi-managed service:

  • Easy installation
  • Multiple Kafka clusters
  • Elastic scaling of brokers
  • Replication and graceful shutdown for high availability
  • Kafka cluster and broker monitoring

Features

DC/OS Kafka provides the following features:

  • Single-command installation for rapid provisioning
  • Multiple clusters for multiple tenancy with DC/OS
  • High availability runtime configuration and software updates
  • Storage volumes for enhanced data durability, known as Mesos Dynamic Reservations and Persistent Volumes
  • Integration with syslog-compatible logging services for diagnostics and troubleshooting
  • Integration with statsd-compatible metrics services for capacity and performance monitoring

Related Services

Install and Customize

Kafka is available in the Universe and can be installed by using either the web interface or the DC/OS CLI.…Read More

Kerberos

In order to run Apache Kafka with Kerberos security enabled, a principal needs to be added for every broker in the cluster. For example, a three node cluster with the default service primary (service.security.kerberos.primary) of kafka will require to following principals:…Read More

SSL Auth

…Read More

Uninstall

If you are using DC/OS 1.10 and the installed service has a version greater than 2.0.0-x:…Read More

Quick Start

Install a Kafka cluster. If you are using open source DC/OS, install a Kafka cluster with the following command from the DC/OS CLI. If you are using Enterprise DC/OS, you may need to follow additional instructions. See the Install and Customize section for more information.…Read More

Connecting Clients

Through Confluent:…Read More

Managing

You can make changes to the service after it has been launched. Configuration management is handled by the scheduler process, which in turn handles deploying DC/OS Kafka Service itself.…Read More

API Reference

…Read More

Troubleshooting

The Kafka service will be listed as “Unhealthy” when it detects any underreplicated partitions. This error condition usually indicates a malfunctioning broker. Use the dcos beta-kafka topic under_replicated_partitions and dcos beta-kafka topic describe commands to find the problem broker and determine what actions are required.…Read More

Limitations

The “disk” configuration value is denominated in MB. We recommend you set the configuration value log_retention_bytes to a value smaller than the indicated “disk” configuration. See the Configuring section for instructions for customizing these values.…Read More

Supported Versions

…Read More

Release Notes

This is a beta release of the DC/OS Apache Kafka service. It contains multiple improvements as well as new features that are to be considered of beta quality. Do not operate this version in production.…Read More

Upgrade

We support upgrade/rollback between adjacent versions only. Concretely, to upgrade from version 2 to version 4, you must upgrade from 2 -> 3, then from 3 -> 4.…Read More