Teams that decide to operate an Apache Kafka cluster often decide in favor of running it inside of Kubernetes, mostly because the engineers are already familiar with Kubernetes. Kubernetes brings some significant advantages that make it easy to collect log messages & metrics, automatically scale and in general manage Docker containers. On the other hand setting up and running a stateful and distributed system such as Kafka is not trivial.

A common question that comes up is how Kafka should be operated in Kubernetes. The three considered options usually are:

  1. Writing all Kubernetes manifests in YAML by hand
  2. Using an existing Helm chart
  3. Using a Kafka Operator (and if so; what operator should I use?)

This article breaks down the pros and cons of each approach and is supposed to help you with making a reasonable decision that suits your requirements.

Writing Kubernetes Manifests

I'm a fan of staying in control of a system and knowing every bit that is happening, so that I'm also able to troubleshoot (and fix) most issues as they occur. Additionally I made some bad experience using a Kubernetes Operator for Prometheus 2 years ago, that caused issues in my environment that I couldn't solve without a patch for the operator.

Because of these two reasons I often prefer defining all the Kubernetes resources on my own (willingly knowing that this is quite some effort). I can utilize the newest Kubernetes features, I have full control of all Kubernetes resources and I'm able to configure everything in Kafka as I like to. But with great power comes great responsibility. You are not only in risk of misconfiguring your Kubernetes resources, you have to solve three challenges that require at least some tricks:

Challenge 1: Configuring the server properties

The server.properties file in Kafka defines all configuration options for the Kafka broker. While you can configure all these options via command line arguments as well (thanks to kafka-server-start.sh), you still have to configure certain options dynamically for each broker, e.g.:

broker.id=0
broker.rack=europe-west1-c
advertised.listeners=SASL_SSL://broker-0.mycompany.com:9092

Because Kubernetes sets the pod's name as env variable HOSTNAME you can do a little trick to configure the broker.id and advertised.listeners dynamically, by using a start command like this:

/opt/kafka/bin/kafka-server-start.sh \
  --override broker.id=${HOSTNAME##*-}
  --override advertised.listeners=SASL_SSL://broker-${HOSTNAME##*-}.mycompany.com:9092

However if you think about configuring the broker.rack correctly, things get trickier. We want to set the rack to the zone the Kafka pod is running in. Because the zone information is tied to a Kubernetes node (set as node label), we have to propagate the node label value into the pod somehow. Ideally we could simply make any node label available to the pod via an environment variable, but unfortunately that's not yet possible with Kubernetes (see this GitHub issue). What we can do instead is to expose the node name as environment variable and on startup the Kafka pod will run a script that talks to the Kubernetes master. The script will describe the node in order to get the value of the availablity zone label (e.g topology.kubernetes.io/zone). The label value will then be used to configure the broker.rack.

Solsson from Yolean maintains the repository Yolean/kubernetes-kafka and shows how it can be done using a bash script:

  1. The startup script: kubernetes-kafka/kafka/10broker-config.yml
  2. The init container: kubernetes-kafka/kafka/50kafka.yml

Huge shoutout to Staffan for maintaining this repository in recent years. You can find lots of valuable Kafka ops know-how and inspiration in there.

Challenge 2: Proper Readiness / Startup Probes

Most community repositories suggest to simply use the tcpSocket check against the port from a Kafka listener (e.g. port 9092). The problem with this probe is that it doesn't actually indicate whether a broker is done with the startup routine. If you run Kafka in a StatefulSet and use the default updateStrategy=RollingUpdate you can quickly end up with two offline brokers because one starting broker may mark itself as ready too early.

You can avoid the problem if you switch the updateStrategy to OnDelete and then take care of restarting the brokers manually one by one. Because these manual restarts quickly become cumbersome you have at least two options to implement a proper readiness probe:

  1. Check the broker's state via JMX: The MBean kafka.server:type=KafkaServer,name=BrokerState exports the BrokerState Enum. If this reports 3 it means the broker is in state BrokerRunning and therefore the broker can be considered ready. In practice the probe can be implemented by writing a small bash script that executes kafka-run-class.sh (part of Kafka) which again runs the class kafka.tools.JmxTool and then checks the response for the reported BrokerState. Here's the full example: kloyan/kafka-startup-probe.sh
  2. Send an API Versions request to the local broker: The Kafka protocol defines a ApiVersionsRequest that clients can use to obtain the version ranges of all requests supported by the broker. It does not require authentication and it doesn't involve other brokers or metadata to respond to this request. This makes it suitable to use as request to check whether a broker is ready to serve requests. If you connect against a listener that uses SSL, you have to configure the client that sends the request accordingly.

    readinessProbe:
        initialDelaySeconds: 10
        timeoutSeconds: 5
        exec:
            command:
            - sh
            - -c
            - "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server=localhost:9092"

Challenge 3: Restart order within a StatefulSet

For automatic rolling restarts you can only rely on the restart order Kubernetes offers. It does restart the pod with the highest ordinal index first and goes one-by-one until it has restarted the pod with the index 0. Ideally you would first restart the Kafka broker, that is the designated controller in the cluster. This avoids unnecessary leader elections. It will usually not cause any issues if you don't do so, but this "best practice" is a good example to illustrate the limitations with plain Kubernetes resources.

Helm Charts

Because Helm charts are not much more than an abstraction layer or templates of the YAML manifests, clusters that are deployed via Helm suffer from the same problems described in the previous section. However some of the challenges (e.g. configuring the broker.id, but not broker.rack) are actually solved in some of the charts: see Bitnami Kafka Chart Scripts.

These charts are pretty complex because they also need to be more generic to cover more potential usecases the chart users may have. This makes it not only hard to understand them, but also the maintenace of such a chart is no longer trivial. If you want to deploy Kafka as quickly as possible into a Kubernetes cluster (e.g. for a POC) and you don't need TLS support, rack awareness etc. using a Helm chart might be a very viable option. Also it is definetely way less effort than writing your own manifests, but you have to give up some fine grained control for that.

Operators

The third common option to deploy Kafka in Kubernetes is by using an operator. Operators have the advantage that they can be quite smart in how they manage the applications in Kubernetes. The user typically just defines the high level configurations such as number of brokers, CPU & RAM requests & limits, storage size, listener configuration, security such as authentication (SASL) and encryption (SSL/TLS) and so on as part of a custom resource. The operator will then take care of actually fulfilling your requirements by creating, watching and updating the Kubernetes resources as needed.

Additionally the operators assist you with certificate management (e.g. by creating all needed certs automatically for the provided CA), simplifying the upgrade process and they solve all the previously described challenges elegantly.

Luckily there are three operators available for Kafka and all of seem to be a decent option as far as I can judge (I have not actually tried all three). So what operator should you choose? In this section I'll try to compare some features and point out the unique selling points, so that you can decide what operator suits your needs the most.

Strimzi

Strimzi is a fully open source, Apache 2.0 licensed Kafka Operator that is a sandbox project in the CNCF since 2019. The CNCF provides Strimzi a vendor-neutral environment and works in favour of the community by acting as independent governance. Five of the six Strimzi maintainers are employed by Red Hat, but Strimzi also regularly receives contributions from the community. Red Hat builds the two commercial products OpenShift Streams and AMQ Streams that are based on Strimzi, which may indicate a long term commitment by the American company.

In terms of features Strimzi does not only support Kafka and ZooKeeper, but also Kafka Connect clusters, MirrorMaker2, Cruise Control and Strimzi Bridge (which is a HTTP API to interact with Kafka). The Strimzi team also developed a so called entity operator, that allows you to manage Kafka Topics and SASL users via custom resources in Kubernetes. Additionally Strimzi utilizes LinkedIn's Cruise Control for operational tasks such as rebalancing, anomaly detection and self-healing within Kafka.

Noteworthy is the excellent and very responsive support via the #strimzi Slack channel in the CNCF server.

Banzai Cloud Operator

Banzai Cloud released their Kafka Operator in the beginning of 2019 and offers two commercial products that are based on this operator: Supertubes Core and Supertubes Pro. This section will not cover any of the commercial features, but if you are curious you can look it up in Banzai Cloud's feature comparison.

Banzai Cloud's operator does also offer Kafka topic and user management via Kubernetes custom resources. However the operator has in general less features compared to Strimzi, but it offers a tight integration with Envoy (for external access via LoadBalancers) and Istio. Via Istio you would get mTLS between all components for free. If you already use Istio and want to integrate your Kafka cluster into the Mesh this operator might be a good choice to achieve that. A few reasons why you want to do that are summarized on the Kafka over Istio page at Banzai Cloud.

Confluent for Kubernetes (CFK)

Confluent for Kubernetes (supersedes the Confluent Operator) is the commercial operator from Confluent. Gwen Shapira from Confluent gave an insight at the Kafka Summit Europe 2021 about how Confluent Cloud is built and she explained that their hosted clusters are also operated via an operator. The internally used operator is in terms of automation & features the same code as offered with CFK. Accordingly it is safe to assume that the the operator is battle tested against thousands of Kafka clusters, which enabled Confluent to make many learnings that have been incorporated to create a more robust operator.

CFK is the only operator that has support for Confluent's enterprise products which makes it the obvious choice for Confluent customers. It supports the whole Confluent Platform suite that comes with paid features such as tiered storage, self-balancing clusters, RBAC, cluster linking, control center and schema validation on the broker.

Conclusion - When to use what

Kafka is a stateful, distributed system that is not trivial to operate on Kubernetes. If you decide to write your own manifests or use a Helm chart to deploy Kafka you need to solve a few challenges that require some custom code/scripts in order to get a production ready, highly available Kafka cluster. Only teams that require the full control about every piece and bit of the deployment, have a lot Kubernetes & Kafka knowledge should consider this way of deploying Kafka. The additional effort should not be underestimated though, especially if you need to run many (i.e. more than 3) Kafka clusters.

The Kubernetes operator pattern is a very good fit for running Kafka on Kubernetes as it makes the operation part very easy. Thanks to the large Kafka ecosystem users can decide between two open source options and the commercial operator from the Kafka creators at Confluent. All these operators are backed by large tech companies, that built a business around their operators and were likely built to stay for a while :-). Kafka Operators do not only help you with a much easier setup, operational tasks (such as certificate management), user & topic management or the dynamic configuration: The biggest advantage is that these operators are running thousands of Kafka clusters and all the learnings made across these clusters will be incorporated so that other users can profit from that. This will save many users from various Kafka issues in the wildest edge cases. The existing operators look quite mature nowadays and they are likely to become just better and will provide more features as the time progresses.