Connecting to AWS S3 Data Lakes Using REST

In PubSub+ event broker version 10.1.0 and later, you can configure a REST Delivery Point (RDP), to extend and produce messages into AWS S3 Data Lakes.

Goals

The goal of this tutorial is to demonstrate how to set up a REST connector in a PubSub+ event broker to send messages to AWS S3 Data Lakes from a remote connection. This tutorial shows you how to:

  • Configure the event broker to set up RDPs with a REST connector.
  • Send messages to an AWS S3 Data Lakes bucket using the Try Me! functionality built intoPubSub+ Broker Manager.
  • Review messages in the AWS S3 Data Lakes objects list.

Connecting to AWS S3 Data Lakes

The REST Connector Wizard can help you set up and configure integrations to various cloud platforms and their services.

This example shows how to integrate the PubSub+ event broker with AWS S3 Data Lakes.

  1. Log into PubSub+ Broker Manager.
  2. Launch the Connector Wizard (see Creating REST Delivery Points).

  3. Click Amazon AWS.

  4. Click AWS S3 then click Authenticate.

  5. Provide a name for the connector and enter the Access Key ID, Access Key, Bucket Name, and Bucket Region for your AWS S3 Data Lakes bucket.

    You can get the Access Key ID, Access Key, Bucket Name, and Bucket Region from your AWS S3 console.

  6. Click Connector Setup and configure the client profile accordingly.

  7. Click Subscription Setup and set up the subscription for the messages. In this example a file name mapping function is used to convert messages to file names with a substitution expression that appends the current time stamp to the file name.

  8. Click Create and Enable Connector to finalize the connector creation.

  9. Use the PubSub+ Broker Manager Try Me! application to send a message. Note that the topic the message is published to matches the subscription set up previously.

  10. View the message in the Amazon S3 Data Lakes objects list as per the file name mapping function used above. In this case the message is a file named with the message time stamp in the test-aws-s3 folder.