Connecting to Apache Kafka with Click-to-Connect

In PubSub+ Broker Manager, you can use the click-to-connect wizard to create an integrated Kafka bridge in a holistic way that provisions all common portions of the service and connection.

Refer to Kafka Bridging for details about how PubSub+ event brokers connect to Apache Kafka.

Prerequisites

  • PubSub+ event broker version 10.8.1 or later, with the appropriate system and Message VPN scaling parameters configured to enable Kafka bridging.

  • Access to PubSub+ Broker Manager (refer to Accessing Broker Manager).

  • If you are setting up a REST connector for a PubSub+ Cloudevent broker services, you need to generate an API token to allow the Click-to-Connect wizard to authenticate with the event broker service. For more information, see Creating an API Token for Connector Wizards.

Receiving from Apache Kafka

To configure an event broker to receive messages from Apache Kafka using the click-to-connect wizard.

  1. Log in to PubSub+ Broker Manager.

  2. Select a Message VPN.

  3. In the navigation bar, click Connector Wizards.

  4. Click Apache.

  5. Click Receive From Apache Kafka then click Authenticate.

  6. Provide a name for the connector and select the Authentication Scheme. This is the scheme the event broker uses to establish a connection to the remote Kafka cluster. Based on the scheme you selected, provide the following:

    • None—no authentication required. This option may be useful for anonymous connections or when a Kafka receiver does not require authentication.

    • Basic—username and password.

    • SCRAM—username, password, and the SCRAM hash.

    • Client Certificate—the PEM formatted content for the client certificate and the password for the certificate.

    • Kerberos—the service name of the remote Kafka broker, the user principal name of the Kafka receiver, and the keytab file for the Kafka receiver.

    • OAuth—the OAuth client ID, client secret, and token endpoint URL. You can also specify the OAuth scope.

    For more information about these authentication schemes, see Configuring Authentication Schemes for Kafka Receivers.

  7. Click Connector Setup then provide the Bootstrap IP Address list. A bootstrap address is the fully qualified domain name (FQDN) or IP address and optional port of one Kafka broker in a Kafka cluster where the Kafka receiver can fetch the state of the entire cluster. You can configure a list of these addresses for the Kafka receiver to try in the event that an attempt to connect to one address fails.

  8. Optional. Configure TLS/SSL encryption. For more information, see Enabling TLS/SSL Encryption for Kafka Receiver Connections.

  9. Click Topic Setup and then provide the name of the Kafka Topic you want to receive messages from, and specify the Initial Offset to consume from that topic.

  10. Specify Solace Topic Mapping. You do this by providing a substitution expression that generates the SMF topic for each message received from Kafka. For more information about substitution expressions, see Substitution Expressions Overview. For example, you could specify the substitution expression rectopic1, so all messages the Kafka receiver consumes from the remote Kafka cluster are published to this topic on the event broker.

  11. Optional. Specify Solace Key Mapping. You do this by providing a substitution expression that generates a partition key for each message received from Kafka. This key determines which queue partition a message is sent to.

  12. Click Create and Enable Connector to finalize the connector creation.

Sending to Apache Kafka

To configure an event broker to send messages to Apache Kafka using the click-to-connect wizard.

  1. Log in to PubSub+ Broker Manager.

  2. Select a Message VPN.

  3. In the navigation bar, click Connector Wizards.

  4. Click Apache.

  5. Click Send To Apache Kafka then click Authenticate.

  6. Provide a name for the connector and select the Authentication Scheme. Based on the scheme you selected, provide the following:

    • None—no authentication required. This option may be useful for anonymous connections or when a Kafka receiver does not require authentication.

    • Basic—username and password.

    • SCRAM—username, password, and the SCRAM hash.

    • Client Certificate—the PEM formatted content for the client certificate and the password for the certificate.

    • Kerberos—the service name of the remote Kafka broker, the user principal name of the Kafka sender, and the keytab file for the Kafka sender.

    • OAuth—the OAuth client ID, client secret, and token endpoint URL. You can also specify the OAuth scope.

    For more information about these authentication schemes, see Configuring Authentication Schemes for Kafka Senders.

  7. Provide the Bootstrap IP Address list.

  8. Optional. Configure TLS/SSL encryption. For more information, see Enabling TLS/SSL Encryption for Kafka Sender Connections.

  9. Click Subscription Setup and provide the name of the Kafka Topic on the remote Kafka cluster that you want to send messages to.

  10. Optional. Specify Kafka Key Mapping . You do this by providing a substitution expression that generates the Kafka partition key for each message sent to Kafka. For more information about substitution expressions, see Substitution Expressions Overview.

  11. Add a list of topic subscriptions from which messages are drawn and sent to the remote Kafka cluster.

  12. Click Create and Enable Connector to finalize the connector creation.