Kafka Bridging Example

The following sections provide an example of how you can use Kafka bridging to exchange messages between a Solace event broker and a Kafka cluster.

Any Kafka bridging configuration made in BETA versions (earlier than 10.6.1) used in standalone deployments will be discarded upon upgrade to version 10.6.1 and later. In addition, you must remove any BETA Kafka bridging configuration used in a redundant (HA) deployment before attempting to upgrade to 10.6.1 or later.

Before You Begin

Before you start this example, you must have:

a Kafka cluster with basic username and password authentication configured.
a configured Message VPN.
a queue (q2 in this example) with appropriate topic subscriptions configured in the Message VPN.
the appropriate system and Message VPN scaling parameters configured to enable Kafka bridging. For more information, see Maximum Number of Kafka Broker Connections and Configuring the Maximum Number of Kafka Broker Connections.

Step 1: Configure a Kafka Receiver

In this example, the first step is to configure a Kafka receiver on the event broker so it can consume events from the remote Kafka cluster.

To configure a Kafka receiver:

Create a Kafka receiver. In this example, the Kafka receiver named receiver1 is configured on the kafka-bridging-vpn Message VPN.

solace> enable
solace# configure
solace(configure)# message-vpn kafka-bridging-vpn
solace(configure/message-vpn)# kafka
solace(configure/message-vpn/kafka)# create kafka-receiver receiver1

Configure the bootstrap address list. In this example, at least one Kafka broker in the remote Kafka cluster can be found at kafka.example.com.
```
solace(...gure/message-vpn/kafka/kafka-receiver)# bootstrap-addresses kafka.example.com
```
Select an authentication scheme. This example uses a basic username and password combination to authenticate with the remote Kafka cluster.
```
solace(...gure/message-vpn/kafka/kafka-receiver)# authentication auth-scheme basic
```

Configure the authentication scheme.

solace(...gure/message-vpn/kafka/kafka-receiver)# authentication basic username example-user1 password example-password1

Optional. Configure the consumer group. This example sets the consumer group ID for the Kafka receiver to cg1.
```
solace(...gure/message-vpn/kafka/kafka-receiver)# group id cg1
```
Create a topic binding. In this example, the Kafka receiver consumes events from a topic named topic1 configured on the remote Kafka cluster.
```
solace(...gure/message-vpn/kafka/kafka-receiver)# create topic-binding topic1
```
Configure the topic binding. In this example, all events the Kafka receiver consumes from the remote Kafka cluster will be published to a topic named rectopic1 on the event broker.
```
solace(...pn/kafka/kafka-receiver/topic-binding)# local topic rectopic1
```

Enable the topic binding.

solace(...pn/kafka/kafka-receiver/topic-binding)# no shutdown

Enable the Kafka receiver.

solace(...pn/kafka/kafka-receiver/topic-binding)# exit
solace(...gure/message-vpn/kafka/kafka-receiver)# no shutdown

Step 2: Configure a Kafka Sender

Next, we will configure a Kafka sender on the event broker so that it can send messages to the remote Kafka cluster.

To configure a Kafka sender:

Create the Kafka sender. In this example, the Kafka sender named sender1 is configured on the kafka-bridging-vpn Message VPN.

solace> enable
solace# configure
solace(configure)# message-vpn kafka-bridging-vpn
solace(configure/message-vpn)# kafka
solace(configure/message-vpn/kafka)# create kafka-sender sender1

Configure the bootstrap address list.

solace(...gure/message-vpn/kafka/kafka-sender)# bootstrap-addresses kafka.example.com

Select an authentication scheme.

solace(...gure/message-vpn/kafka/kafka-sender)# authentication auth-scheme basic

Configure the authentication scheme.

solace(...gure/message-vpn/kafka/kafka-sender)# authentication basic username example-user1 password example-password1

Optional. Configure idempotence.

solace(configure/message-vpn/kafka/kafka-sender)# idempotence

Create a queue binding. In this example, the Kafka sender is bound to a queue named q2 configured on the event broker.
```
solace(configure/message-vpn/kafka/kafka-sender)# create queue-binding q2
```
Configure the queue binding. In this example the Kafka sender expects acknowledgements from all replicas on the Kafka cluster, contains a consistent partition scheme that selects a Kafka partition based on a CRC hash of the partition key generated by the binding (in this case key0). In addition, all messages are sent to a topic named topic2 on the remote Kafka cluster.
```
solace(...-vpn/kafka/kafka-sender/queue-binding)# ack-mode all
solace(...-vpn/kafka/kafka-sender/queue-binding)# partition scheme consistent
solace(...-vpn/kafka/kafka-sender/queue-binding)# partition consistent hash crc
solace(...-vpn/kafka/kafka-sender/queue-binding)# remote key key0
solace(...-vpn/kafka/kafka-sender/queue-binding)# remote topic topic2
```

Enable the queue binding.

solace(...-vpn/kafka/kafka-sender/queue-binding)# no shutdown

Enable the Kafka sender.

solace(...-vpn/kafka/kafka-sender/queue-binding)# exit
solace(configure/message-vpn/kafka/kafka-sender)# no shutdown

With this configuration, you can now send messages to and receive events from your Kafka cluster.

Provide feedback