Getting Started With the Connector for Snowflake

Presuming you’re using the default application.yml file provided within the download, follow one of the quick starts below to connect a PubSub+ event broker and Snowflake. The quick starts use specified credentials as examples to get started with two workflows enabled, workflow 0 and workflow 1.

Where:

  • Workflow 0 consumes messages from the SolacePubSub+ queue, Queue-A and publishes them to the Snowflake table, DEMO.PUBLIC.TABLEA.

  • Workflow 1 consumes messages from the SolacePubSub+ queue, Queue-B and publishes them to the Snowflake table, DEMO.PUBLIC.TABLEB.

A workflow is the configuration of a flow of messages from a source to a target. The connector supports up to  20 individual, concurrent workflows  per instance.

The connector does not provision Solace queues that do not exist, but can provision the Snowflake tables based on configuration.

Prerequisites

Quick Start Common Steps

These are the steps required to run all quick-start examples:

  1. Configure a Snowflake user account with a public/private key pair for authentication.

  2. Grant the necessary permissions to the user account to allow for table creation and data insertion.

  3. (Optional) Create the desired target test tables in accordance with the standard connector schema.

  4. Update the provided samples/config/application.yml with the credentials and entities for your deployment.

Quick Start: Running the Connector Using the Command Line

Run:

java -jar pubsubplus-connector-snowflake-<version>.jar --add-opens=java.base/java.nio=ALLUNNAMED
--spring.config.additional-location=file:samples/config/

Where:

  • <version> is the version of the connector you are using, such as 2.0.0.

By default, this command detects any Spring Boot configuration files as per the Spring Boot's default locations.

For more information, see Configure Locations to Find Spring Property Files.

Quick Start: Running the Connector Using the start.sh Script

For convenience, you can start the connector through the shell script using the following command:

chmod 744 ./bin/start.sh
./bin/start.sh [-n NAME] [-l FOLDER] [-p PROFILE] [-c FOLDER] [-ch HOST] [-cp PORT] [
-j FILE] [-cm] [-cmh HOST] [-cmp PORT] [-mh HOST] [-mp PORT] [-o OPTIONS] [-b]

If you use invalid parameters, such as in the following example:

./bin/start.sh -l dummy_folder -c dummy_folder -j dummy_file.jar

The script shows you all errors at the same time:

Solace PubSub+ Connector for Snowflake		

Connector startup failed:

Following folder doesn't exists on your filesystem:     'dummy_folder'		
Following folder doesn't exists on your filesystem:     'dummy_folder'
Following file doesn't exists on your filesystem:       'dummy_file.jar'

In situations where you have don't provide a parameter, the script runs with the predefined values as follows:

The default name for the JAR file is pubsubplus-connector-snowflake-2.0.0.jar.

Parameter Default Value Description
-n, --name application The name of the connector instance, that is configured in [spring.application.name]. This name impacts the grouping of connectors only.

-l, --libs

./libs

The directory that contains the required and optional dependency JAR files, such as Micrometer metrics export dependencies (if configured). If this option is not specified, it will use the current ./libs/ directory.

-p, --profile

empty, no profile is used

The profile to be used with the connector’s configuration. The configuration file named 'application-<profile>.yml' is used. If  this option is not specified, no profile is used.

-c, --config

./ or current folder

The path to the folder containing the configuration files to be applied when the connector starts up for chosen profile. If not specified, the current directory is used.
-H, --host 127.0.0.1 Specifies the host where the connector runs.
-P, --port 8090 Specifies the port where connector runs.
-mp, --mgmt_port 9009 Specifies the management port for back calls of current connector from PubSub+ Connector Manager. This parameter is ignored if the -cm parameter is not provided.

-j, --jar

The name of the JAR file. See JAR file for this connector.

The path to the specified JAR file to start the connector. If the option is not specified, the default JAR file is used from the current directory.
-cm, --manager application

Specifies Connector Managerto use the configuration storage and allows you to enable the cloud configuration for the connector. When this parameter is enabled, you can specify the -mp or --mgmt_port, -H, or --host, and -cmh with the -cmp parameters, unless you want to use the default values for those parameters.

This option disables the listed parameters to be read from the configuration file. In this case, the operator must explicitly specify the parameters for the script, otherwise default values are used.

-cmh, --cm_host  

 127.0.0.1 Specifies the host where Connector Manager is running. This parameter is ignored if -cm is not provided.
-cmp, --cm_port  9500 Specifies the port where Connector Manager is running. This parameter is ignored if -cm is not provided.

-o, --options

No default values

Specifies the JVM options used on when the connector starts. For example, -Xms64M -Xmx1G.
-tls  N/A Specifies to use HTTPS instead of HTTP. When this parameter is used, the configuration file must contain an additional section with the preconfigured paths for the key store and trust store files.
-s, --show N/A Performs a dry run (does nothing). The output prints the start CLI command and its raw output and exits. This parameter is useful to check your parameters without running the connector.

-b, --background

N/A

Runs the connector in the background. No logs are shown and the connector continues running in detached mode.

-h, --help

N/A

Prints help information and exits.

The script can provide more help information from the command line by using the -h option. For more configuration examples of starting the connector together with Connector Manager, see the examples provided in the Connector Manager archive.

Quick Start: Running the Connector as a Container

The following steps show how to use the sample docker compose file that has been included in the package:

  1. Change to the docker directory:

    cd samples/docker

    This directory contains both the docker-compose.yml file, as well as an .env file that contains environment secrets required for the container’s health check.

  1. Run the connector:

    docker-compose up -d

    The sample docker compose file:

    • Exposes the connector’s 8090 web port to 8090 on the host.

    • Connects a PubSub+ event broker and the Snowflake exposed on a host using default ports.

    • Mounts the samples/config directory.

    • Mounts the previously defined libs directory.

    • Creates a healthcheck user with read-only permissions.

      • The default username and password for this user can be found within the .env file.

      • This user overrides any users you have defined in your application.yml. See Security for more information.

    • Uses the connector’s management health endpoint as the container’s health check.

For more information about how to use and configure this container, see Modifying the Container Configuration.