Kafka Bridging

Kafka bridging allows for the configuration of two objects to bridge between a Kafka cluster and a Solace PubSub+ software event broker. Once configured, message flow is possible in both directions. This behavior is directly embedded in the event broker, in other words, no external Kafka Connect infrastructure is required in order to pass messages to and from Kafka.

A Kafka bridge consists of the following configuration objects:

• A Kafka Receiver —receives events from one or more Kafka topics, converts the events to Solace Message Format (SMF) messages, and publishes them to topics on the PubSub+ event broker.

• A Kafka Sender—takes SMF messages from one or more queues, converts the messages to Kafka events, and sends them to Kafka topics on the remote Kafka cluster.

This feature is only supported on software event brokers, and not appliances.

Kafka Receiver

Each Kafka receiver contains a list of topic bindings. Each topic binding names the Kafka topic that messages are drawn from, and includes attributes that dictate how messages from the Kafka topic are translated into a Solace message.

Messages translated from a Kafka topic are published into the Solace message bus. Because a Kafka topic may not be appropriate as a Solace topic, you must use a substitution expression to construct a Solace topic for the received message. Substitution expressions are a Solace-specific language used to replace specific text attributes with system generated output. A common pattern in Kafka topics, for example, is to embed a hierarchy in them using dots or underscores (e.g. "a.b.c"), and an expression such as ${replace(kafkaTopic(),".","/")} would convert these to an SMF topic and allow for more rich topic matching once within the Solace network.

These components are illustrated in the following diagram:

The following table describes how messages are converted from Kafka brokers to PubSub+ event brokers:

Kafka	Solace	Comments
payload	payload	The payload is always written to the Solace binary attachment.
partition key	partition key	The Solace partition key is set based on how you configure the topic binding's local key substitution expression. For more information, see Configuring Partition Key Generation.
topic	user property k_topic
partition number	user property k_partition
partition offset	user property k_offset
timestamp	sender timestamp
headers	user properties	All Kafka headers are converted to Solace user properties.
<various>	topic	The Solace topic is set based on how you configure the topic binding's local topic substitution expression. For more information, see Configuring SMF Topic Generation.

For information about how to set up an event broker to receive events from a Kafka cluster using a Kafka receiver, see Configuring a Kafka Receiver.

Kafka Sender

Each Kafka sender contains a list of queue bindings. Each queue binding names the queue that messages are drawn from, and includes attributes that dictate how those messages are sent to a Kafka cluster. These are ordinary queues, configured outside of the context of the Kafka sender, and as such the messages that the queue attracts depend on the set of queue subscriptions configured for it. These messages are then sent to a single Kafka topic configured per queue binding.

Because Kafka topics are partitioned, with partition choice being determined by the client (in this case the Solace event broker), each Kafka sender offers several partition choice options:

• Random—select a random partition of the Kafka topic.

• Explicit—select an explicit partition of the Kafka topic.

• Consistent—use a hash of a partition key to select a partition.

When using a consistent partition selection scheme, you can use a substitution expression to determine the partition key . For example, you could use the expression ${topic()}, which would generate a partition key which matches the published Solace topic. This would result in all messages with the same Solace topic being sent to the same Kafka topic partition.

A substitution expression can be used to generate a key for the Kafka message even if a consistent partition scheme is not being used. There might be other entities in the Kafka cluster that use the key, and being able to set it using a substitution expression can be useful, even if the Kafka sender doesn't use it to determine the partition.

These components are illustrated in the following diagram:

The following table describes how messages are converted from Solace PubSub+ event brokers to Kafka brokers:

Solace	Kafka	Comments
payload	payload
	topic	The Kafka topic is set based on how you configure the queue binding's remote topic attribute. This is a single fixed value per queue binding. For more information, see Configuring SMF to Kafka Topic Mapping
<various>	partition key	The Kafka partition key is generated per-message based on how you configure the queue binding's remote key substitution expression. For more information, see Configuring Kafka Partition Key Generation.
sender timestamp	timestamp
user properties	headers	All Solace user properties are converted to Kafka headers.

For information about how to set up the event broker to send messages to a Kafka cluster using a Kafka sender, see Configuring a Kafka Sender.