Redpanda is a modern event streaming platform that aims to be fully compatible with Apache Kafka's API. It promises to be more performant, operational simplicity and safer defaults to avoid data loss or downtimes. Since Redpanda claims to be compatible with Kafka's API we were immediately hooked by the idea and thought it's worth giving a shot.

In fact it turned out to be a no brainer - as promised by the Redpanda devs. Except for one small change we didn't need to change anything to be fully compatible with Redpanda. The required change is due to the fact that Redpanda publishes a control record right after topic creation in order to proof that a raft quorum exists. This however is completely on par with the Kafka protocol and therefore is more likely an issue within Kowl. More common consumer/producer applications should not notice a difference at all.

Test Redpanda with Kowl in Docker

Since Redpanda offers officially maintained Docker images we can just use their latest docker image (at the time of writing this post it is vectorized/redpanda:v21.4.12) and create a simple docker-compose file which looks like that:

version: '3.7'
      - /usr/bin/rpk
      - redpanda
      - start
      - --smp
      - '1'
      - --reserve-memory
      - 0M
      - --overprovisioned
      - --node-id
      - '0'
      - --kafka-addr
      - --advertise-kafka-addr
      - PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
    image: vectorized/redpanda:v21.4.12
      - 9092:9092
      - 29092:29092
      - KAFKA_BROKERS=redpanda:29092
      - 8080:8080
    depends_on: [ redpanda ]
      - SHOP_KAFKA_BROKERS=redpanda:29092
    depends_on: [ redpanda ]

We spin up three containers:

  • Redpanda
  • Kowl
  • OwlShop (emulates a full ecommerce shop by producing data to Kafka topics and consuming the data via consumer groups)

You also might wonder why there is no ZooKeeper container: This is because Redpanda does not require ZooKeeper, it utilizes the Raft consensus algorithm instead. The deprecation of ZooKeeper in favour of Raft is also planned for Apache Kafka with KIP-500. Once you started the docker-compose file via docker-compose up you can visit http://localhost:8080 to browse the cluster's topics which should return some demo data produced by the OwlShop.


My impression is that Redpanda in general is a promising, more modern option to Kafka. Redpanda is not yet 100% API compatible with the latest Kafka versions, but it has quite some advantages to offer - performance and operational simplicity being the most prominent ones. Even though Kafka just released it's first experimental version with ZK Removal (see v2.8.0 release notes), it is fair to point out that this is different than Redpanda's architecture. KIP-500 effectively replaces ZooKeeper with a quorum service and therefore you still have two different distributed systems (quorum service and Kafka) with their own fault domains. Redpanda on the other hand is a single distributed system.

What I like about Redpanda is that it seems to be more accessible for the broad community: It utilizes GitHub for issue and feature tracking, Slack for community support (devs are very active there!), native Prometheus support, safe defaults and I think it delivers the promise of operational simplicity:

  • No ZooKeeper required
  • No config tuning (neither for performance nor safety) required
  • Native Prometheus support
  • Official Kubernetes Operator for free

The hype around Redpanda shows that they address some painpoints, now we have to see how well it's going to be adopted in the real world.

kowl redpanda chillin
Redpanda and Kowl being happy