Connecting to AWS S3 Data Lakes Using REST

In the 10.1.0 release of the PubSub+ event broker, you can configure a REST delivery point (RDP), to extend and produce messages into AWS S3 Data Lakes.

Goals

The goal of this tutorial is to demonstrate how to set up a REST connector in a PubSub+ event broker to send messages to AWS S3 Data Lakes from a remote connection. This tutorial will show you how to:

  • Configure the event broker to set up RDPs with a REST connector
  • Send messages to an AWS S3 Data Lakes bucket using the Try Me! functionality built into the PubSub+ Broker Manager
  • Review messages in the AWS S3 Data Lakes objects list

Prerequisites

  • Solace PubSub+ Event Broker 10.1.0 or higher
  • Access to PubSub+ Broker Manager (refer to PubSub+ Broker Manager)
  • A preconfigured AWS S3 Data Lakes bucket

Connecting to AWS S3 Data Lakes

The REST Connector wizard can help you set up and configure integrations to various cloud platforms and their services.

This example shows how to integrate the PubSub+ event broker with AWS S3 Data Lakes.

  1. Log into the PubSub+ Broker Manager.
  2. Launch the connector Click-to-Connect Wizard (refer to Creating a REST Connector with Click-to-Connect).

  3. Click Amazon AWS.

  4. Click AWS S3 then click Authenticate.

  5. Provide a name for the connector and enter the Access Key ID, Access Key, Bucket Name, and Bucket Region for your AWS S3 Data Lakes bucket.

    You can get the Access Key ID, Access Key, Bucket Name, and Bucket Region from your AWS S3 console.

  6. Click Connector Setup and configure the client profile accordingly.

  7. Click Subscription Setup and set up the subscription for the messages. In this example a file name mapping function is used to convert messages to file names with a substitution expression that appends the current time stamp to the file name.

  8. Click Create and Enable Connector to finalize the connector creation.

  9. Use the PubSub+ Broker Manager Try Me! application to send a message. Note that the topic the message is published to matches the subscription set up previously.

  10. View the message in the Amazon S3 Data Lakes objects list as per the file name mapping function used above. In this case the message is a file named with the message time stamp in the test-aws-s3 folder.