Relay Kafka

In this guide, we will cover how to relay a Kafka topic to Streamdal using Plumber. This is particularly useful when you want to migrate data, forward messages, or transform data in transit from Kafka to Streamdal.

Prerequisites

Before you start, you will need the following:

  • Docker installed on your machine.
  • A running Kafka instance. In this example, we are using Confluent Cloud.
  • Your Streamdal token. You can find this in your Streamdal dashboard under the collection.
  • The Kafka topic you want to relay.

Kafka Relay

To relay a Kafka topic to Streamdal, we will use the plumber relay command in a Docker container. Here’s the Docker command to set up the relay:

docker run -d --name plumber-kafka -p 8080:8080 \
    -e PLUMBER_RELAY_TOKEN="$YOUR-STREAMDAL-TOKEN-HERE"
    -e PLUMBER_RELAY_KAFKA_ADDRESS="pkc-4kgmg.us-west-2.aws.confluent.cloud:9092,pkc-5kgmg.us-west-2.aws.confluent.cloud:9092"
    -e PLUMBER_RELAY_KAFKA_TOPIC="$YOUR_TOPIC"
    -e PLUMBER_RELAY_KAFKA_INSECURE_TLS="true"
    -e PLUMBER_RELAY_KAFKA_USERNAME="$YOUR_CONFLUENT_API_KEY"
    -e PLUMBER_RELAY_KAFKA_PASSWORD="$YOUR_CONFLUENT_API_SECRET"
    -e PLUMBER_RELAY_KAFKA_SASL_TYPE="plain"
    streamdal/plumber plumber relay kafka

Make sure to replace $YOUR-STREAMDAL-TOKEN-HERE, $YOUR_TOPIC, $YOUR_CONFLUENT_API_KEY, and $YOUR_CONFLUENT_API_SECRET with your actual Streamdal token, Kafka topic, Confluent API Key, and Confluent API Secret respectively.

Understanding the Docker Command

Here’s a breakdown of the Docker command:

  • PLUMBER_RELAY_TOKEN: This is your Streamdal token. Plumber uses this to authenticate with Streamdal.
  • PLUMBER_RELAY_KAFKA_ADDRESS: The addresses of your Kafka brokers. If you have more than one broker, separate them with commas.
  • PLUMBER_RELAY_KAFKA_TOPIC: The Kafka topic you want to relay.
  • PLUMBER_RELAY_KAFKA_INSECURE_TLS: Set this to “true” to disable TLS verification. This is not recommended for production environments.
  • PLUMBER_RELAY_KAFKA_USERNAME: Your Confluent API Key.
  • PLUMBER_RELAY_KAFKA_PASSWORD: Your Confluent API Secret.
  • PLUMBER_RELAY_KAFKA_SASL_TYPE: The SASL mechanism to use for authentication. In this case, we are using “plain”.

The plumber relay kafka command tells Plumber to start in relay mode and read from Kafka.

After running this command, Plumber will start relaying messages from the specified Kafka topic to Streamdal.