Connecting to Apache Kafka Using the Connector Wizard

In Broker Manager, you can use a connector wizard to create an integrated bridge with Apache Kafka in a holistic way that provisions all common portions of the service and connection.

For more information about connecting Solace event brokers to Apache Kafka, see Kafka Bridging.

Alternatively, you can set up a Kafka bridge for an event broker service in Broker Manager, without using the connector wizard. For more information, see Configuring Kafka Bridging.

Prerequisites

  • Solace event broker version 10.8.1 or later, with the appropriate system and Message VPN scaling parameters configured to enable Kafka bridging.

  • Access to Broker Manager. For more information, see Using Broker Manager with Event Broker Services.

  • If you are setting up a REST connector for a Solaceevent broker service, you need to generate an API token to allow the connector wizard to authenticate with the event broker service. For more information, see Creating an API Token for Connector Wizards.

Receiving Messages from Apache Kafka

To configure an event broker to receive messages from Apache Kafka using the connector wizard, perform these steps:

  1. Open Broker Manager. For instructions, see Using Broker Manager with Event Broker Services.
  2. On the left navigation bar, click Connector Wizards.
  3. Click Apache.
  4. Click Receive From Apache Kafka then click Authenticate.

  5. Provide a name for the connector and select the Authentication Scheme. This is the scheme the event broker uses to establish a connection to the remote Kafka cluster. Based on the scheme you selected, provide the following:

    • None—no authentication required. This option may be useful for anonymous connections or when a Kafka receiver does not require authentication.
    • Basic—username and password.
    • SCRAM—username, password, and the SCRAM hash.
    • Client Certificate—the PEM formatted content for the client certificate and the password for the certificate.
    • Kerberos—the service name of the remote Kafka broker, the user principal name of the Kafka receiver, and the keytab file for the Kafka receiver.
    • OAuth—the OAuth client ID, client secret, and token endpoint URL. You can also specify the OAuth scope.

    For more information about these authentication schemes, see Creating a Kafka Receiver.

  6. Click Connector Setup then provide the Bootstrap IP Address list. A bootstrap address is the fully qualified domain name (FQDN) or IP address and optional port of one Kafka broker in a Kafka cluster where the Kafka receiver can fetch the state of the entire cluster. You can configure a list of these addresses for the Kafka receiver to try in the event that an attempt to connect to one address fails.
  7. (Optional) Configure TLS/SSL encryption. For more information, see Creating a Kafka Receiver.
  8. Click Topic Setup and then provide the name of the Kafka Topic you want to receive messages from, and specify the Initial Offset to consume from that topic.
  9. Specify Solace Topic Mapping. You do this by providing a substitution expression that generates the SMF topic for each message received from Kafka. For more information about substitution expressions, see Substitution Expressions Overview. For example, you could specify the substitution expression rectopic1, so all messages the Kafka receiver consumes from the remote Kafka cluster are published to this topic on the event broker.
  10. Optional. Specify Solace Key Mapping. You do this by providing a substitution expression that generates a partition key for each message received from Kafka. This key determines which queue partition a message is sent to.
  11. Click Create and Enable Connector to finalize the connector creation.

Sending Messages to Apache Kafka

To configure an event broker to send messages to Apache Kafka using the click-to-connect wizard, perform these steps:

  1. Open Broker Manager. For instructions, see Using Broker Manager with Event Broker Services.
  2. On the left navigation bar, click Connector Wizards.
  3. Click Apache.
  4. Click Send To Apache Kafka then click Authenticate.

  5. Provide a name for the connector and select the Authentication Scheme. Based on the scheme you selected, provide the following:

    • None—no authentication required. This option may be useful for anonymous connections or when a Kafka receiver does not require authentication.
    • Basic—username and password.
    • SCRAM—username, password, and the SCRAM hash.
    • Client Certificate—the PEM formatted content for the client certificate and the password for the certificate.
    • Kerberos—the service name of the remote Kafka broker, the user principal name of the Kafka sender, and the keytab file for the Kafka sender.
    • OAuth—the OAuth client ID, client secret, and token endpoint URL. You can also specify the OAuth scope.

    For more information about these authentication schemes, see Creating a Kafka Sender.

  6. Provide the Bootstrap IP Address list.
  7. Optional. Configure TLS/SSL encryption. For more information, see Creating a Kafka Sender.
  8. Click Subscription Setup and provide the name of the Kafka Topic on the remote Kafka cluster that you want to send messages to.
  9. Optional. Specify Kafka Key Mapping . You do this by providing a substitution expression that generates the Kafka partition key for each message sent to Kafka. For more information about substitution expressions, see Substitution Expressions Overview.
  10. Add a list of topic subscriptions from which messages are drawn and sent to the remote Kafka cluster.
  11. Click Create and Enable Connector to finalize the connector creation.