Creating an Apache Kafka cluster at Aiven is super easy - just a few clicks and a few moments later you have an up to date, production ready Kafka cluster running in a cloud provider & region of your choice. In the Aiven panel you can get an overview about the SASL users, configured ACLs, existing topics, configured connectors and registered schemas.

Now if you want to get more insights about what's going on inside of your Kafka cluster Kowl can help you. Kowl is an open source Web UI that allows you to work more efficiently with Kafka by providing more insights about what's going on in your cluster in the most comfortable way:

  • List all Brokers with their disk usage and full configuration
  • List all topics with their disk usage, config, consumers, replicas, partition details, ...
  • Inspect messages from a topic in a well designed JSON viewer (JSON, Protobuf, Avro, XML, MsgPack supported)
  • Find specific messages in a topic by streaming messages using a JavaScript filter engine (running in the backend)
  • List all ACLs (as of now not possible with Aiven; the used custom authorizer doesn't support listing ACLs)
  • Schema Registry support - list all registered schemas and their versions
  • Reassign partitions (migrate data between brokers, balance partitions across the existing brokers etc)

Short Kowl v1.3 demo

Run Kowl with Aiven

Usually Kowl is supposed to be deployed and centrally managed so that the initial setup has to be done only once. This has the advantage that all your colleagues don't need to bother with the setup, they could just access Kowl via the browser. However for local development and testing Kowl it is possible to run Kowl on your local machine as well.

To run Kowl locally we spin up a docker container that we'll configure via environment variables. We have to specify all connection details including the authentication credentials, which we will be doing step by step now:

Step 1: Download certificates

Download certificates from Aiven
Download certificates from Aiven

New Aiven clusters have configured mutual TLS for authentication by default. If you prefer to use any of the SASL mechanisms for authentication you only need to download the ca.pem file in this step.

Download the certificates that are needed for authentication as well as the ca certificate and save them to a separate directory on your machine. You should now have three files with the following names in your directory:

  • ca.pem
  • service.cert
  • service.key

If the filenames are different you have to rename them or adapt them in the docker command in step 2.

Step 2: Start Kowl container

mTLS Authentication:

Replace the mount path, brokers/schemaregistry connect string, username and password before running this command:

docker run -p 8080:8080 \
--mount type=bind,source=/Users/martin/Downloads/aiven-certs,target=/etc/tls \
-e KAFKA_BROKERS=kafka-4be6020-martin-1296.aivencloud.com:21687 \
-e KAFKA_TLS_ENABLED=true \
-e KAFKA_TLS_CAFILEPATH=/etc/tls/ca.pem \
-e KAFKA_TLS_CERTFILEPATH=/etc/tls/service.cert \
-e KAFKA_TLS_KEYFILEPATH=/etc/tls/service.key \
-e KAFKA_SCHEMAREGISTRY_ENABLED=true \
-e KAFKA_SCHEMAREGISTRY_URLS=https://kafka-4be6020-martin-1296.aivencloud.com:21690 \
-e KAFKA_SCHEMAREGISTRY_USERNAME=avnadmin \
-e KAFKA_SCHEMAREGISTRY_PASSWORD=redacted \
quay.io/cloudhut/kowl:v1.3.1

SASL Plain Authentication:

Replace the mount path, brokers/schemaregistry connect string, username and password before running this command:

docker run -p 8080:8080 \
--mount type=bind,source=/Users/martin/Downloads/aiven-certs,target=/etc/tls \
-e KAFKA_BROKERS=kafka-4be6020-martin-1296.aivencloud.com:21687 \
-e KAFKA_TLS_ENABLED=true \
-e KAFKA_TLS_CAFILEPATH=/etc/tls/ca.pem \
-e KAFKA_SASL_ENABLED=true \
-e KAFKA_SASL_USERNAME=avnadmin \
-e KAFKA_SASL_PASSWORD=redacted \
-e KAFKA_SCHEMAREGISTRY_ENABLED=true \
-e KAFKA_SCHEMAREGISTRY_URLS=https://kafka-4be6020-martin-1296.aivencloud.com:21690 \
-e KAFKA_SCHEMAREGISTRY_USERNAME=avnadmin \
-e KAFKA_SCHEMAREGISTRY_PASSWORD=redacted \
quay.io/cloudhut/kowl:v1.3.1

Run docker image without schema registry enabled
Run docker image without schema registry enabled

Step 3: Open Kowl in your Browser

After you ran the docker command from step 2 you should see a confirmation in the docker logs:

{"level":"info","ts":"2021-04-10T12:19:59.149Z","msg":"Server listening on address","address":"[::]:8080","port":8080}

Now you can visit http://localhost:8080 to open Kowl in your Browser.