Configuring Connection Details

This section provides instructions for configuring the connection details required to establish communication between the Micro-Integration and your third-party system.

For information about configuring the connection to the event broker, see Connecting to Your Event Broker.

This Micro-Integration supports workflows in the following direction:

  • Google Cloud Storage to Solace

For more information about workflows, see Enabling Workflows and Managing Workflows.

The name of the binder for the Google Cloud Storage is gcs.

Solace Connection Details

The Spring Cloud Stream Binder for Solace uses Spring Boot Auto-Configuration for the Solace Java API to configure its session. In the application.yml, this typically is configured as follows:

solace:
  java:
    host: tcp://localhost:55555
    msg-vpn: default
    client-username: default
    client-password: default

For more information and options to configure the Solace session, see Spring Boot Auto-Configuration for the Solace Java API.

Preventing Message Loss when Publishing to Topic-to-Queue Mappings

If the Micro-Integration is publishing to a topic that is subscribed to by a queue, messages may be lost if they are rejected (for example, if queue ingress is shut down).

To prevent message loss, configure the reject-msg-to-sender-on-discard option with the including-when-shutdown flag.

Google Cloud Storage Connection Details

To configure the Google Cloud Storage connection details, set the following values in the application.yml file:

spring:
  cloud:
    stream:
      bindings:
        <inputname>:
          destination: "google-storage://<bucketName>"
          binder: gcs

The Spring Cloud Stream standard properties for the Google Cloud Storage are as follows.

Configuration Option Type Valid Values Description

spring.cloud.stream.bindings.<inputname>.destination

String

A valid Google Cloud Storage bucket name

The Google Cloud Storage bucket name to read from, in the format: 

google-storage://<bucketName>

spring.cloud.stream.bindings.<inputname>.binder

String

gcs

This property must be set to gcs.

Checkpoint Store Configuration

The Micro-Integration for Google Cloud Storage stores the current progress of file processing in a checkpoint store backed by a Last Value Queue (LVQ) on a Solace event broker.

The following table lists the configuration options for the checkpoint store.

Configuration Option Type Valid Values Default Value Description

camel-binder.checkpoint.lvqName

String

Any

None

Required.

The name of the LVQ (spool size 0) to be used for checkpointing. The queue must exist on the same event broker and Message VPN as the target queue.

If the LVQ is deleted (administratively) or a message from the LVQ is deleted or consumed by another consumer, the Micro-Integration will not be able to resume from the last checkpoint. In addition, the LVQ should not be shared by multiple instances of the Micro-Integration.

camel-binder.checkpoint.autoProvisionLvq

boolean

(true | false)

false

Optional.

Set to true to auto-provision the LVQ if it does not exist.

Google Cloud Storage Consumer Configuration Options

The following configuration options are available for Google Cloud Storage consumers. These properties are to be prefixed with spring.cloud.stream.camel.bindings.<inputname>.consumer.endpoint.query-parameters for source bindings.

Configuration Option Type Valid Values Default Value Description

serviceAccountKey

String

A valid File Protocol path to the service account key file

${GOOGLE_STORAGE_SERVICE_ACCOUNT_KEY:}

Required. The service account key in JSON format. It can be loaded by default from classpath, but you can prefix with classpath:, file:, or http: to load the resource from different systems.

deleteAfterRead

Boolean

true, false

false

When true, the Micro-Integration consumes and deletes the file, regardless of whether it has been read before.

When false:

  • If a file has already been consumed, the Micro-Integration periodically downloads it again but ignores it.

  • The time it takes to cycle through the entire folder increases as more files are added.

  • If the Micro-Integration is shut down while a file is pending, it may hang for a short time.

  • Renaming a file will cause it to be consumed again.

The deleteAfterRead property is also configurable as a workflow default by specifying: 

spring.cloud.stream.camel.default.consumer.endpoint.query-parameters.deleteAfterRead.

filter

String

A valid regular expression used to filter objects in the bucket

None

Optional. A regular expression that filters objects in the bucket. Only objects with names matching the regex will be consumed. For example, to consume all files starting with inbound/, use filter: "^inbound/.*".

fileType

String

json, xml, delimited

None

Determines how the connector parses the content of the consumed files. Additional configuration is required. For details about configuring these file types, see one of:

JSON File Configuration Properties

When fileType is set to json, the following configuration options are available:

Configuration Option Type Valid Values Default Value Description

jsonPath

String

A valid JSON path expression

$

(root node)

The JSON path expression to extract, split, or filter the data from the JSON file.

The default value of $ means the entire JSON file content will be processed as a single event.

XML File Configuration Properties

When fileType is set to xml, the following configuration options are available:

Configuration Option Type Valid Values Default Value Description

xPath

String

A valid XPath path expression

/

(root node)

The XPath expression to extract, split, or filter the data from the XML file.

The default value of / means the entire XML file content will be processed as a single event.

Delimited File Configuration Properties

When fileType is set to delimited, the following configuration options are available:

Configuration Option Type Valid Values Default Value Description

eventDelimiter

String

Any character or string

\n

The character or string that separates records in a delimited file.

paramDelimiter

String

Any character or string

,

The character or string that separates fields within each record.

Advanced Configuration Options

The following additional options are available to control the behavior of the Micro-Integration for Google Cloud Storage. These properties are to be prefixed with spring.cloud.stream.camel.bindings.<inputname>.consumer.endpoint.query-parameters.

Option Type Default Description

autoCreateBucket

boolean

true

When true, indicates that the bucket is automatically created.

storageLocation

String

US-EAST1

The Cloud Storage location to use when creating new buckets.

destinationBucket

String

None

The destination bucket where an object must be moved when moveAfterRead is set to true.

downloadFileName

String

None

The folder or filename to use when downloading the blob. By default, this parameter specifies the folder name, and the name of the file is the blob name. For example, setting this parameter to mydownload will be the same as setting mydownload/$\{file:name}. You can use dynamic expressions for fine-grained control. For example, you can specify $\{date:now:yyyyMMdd}/$\{file:name} to store the blob in subfolders based on today's day. Only $\{file:name} and $\{file:name.noext} are supported as dynamic tokens for the blob name.

moveAfterRead

boolean

false

When true, indicates that objects are moved from the origin bucket to a different bucket after they have been retrieved. To accomplish the operation, the destinationBucket option must be set. The copy bucket operation is only performed if the Exchange is committed. If a rollback occurs, the object is not moved.

prefix

String

None

The prefix that is used in the BlobListOptions to consume only objects of interest.

bridgeErrorHandler

boolean

false

When true, indicates that the consumer is bridged to the Camel routing Error Handler, which means any exceptions (if possible) that occur while the Camel consumer is trying to pick up incoming messages will be processed as a message and handled by the routing Error Handler. This is only possible if the third-party component allows Camel to be alerted if an exception was thrown. Some components handle this internally only, and therefore bridgeErrorHandler is not possible. In other situations, the Camel component may be improved to hook into the third-party component and make this possible for future releases. By default, the consumer uses the org.apache.camel.spi.ExceptionHandler to deal with exceptions that will be logged at WARN or ERROR level and ignored.

objectName

String

None

The object name inside the bucket.

backoffErrorThreshold

int

None

The number of subsequent error polls (failed due to some error) that should happen before the backoffMultiplier kicks in.

backoffIdleThreshold

int

None

The number of subsequent idle polls that should happen before the backoffMultiplier kicks in.

backoffMultiplier

int

None

The multiplier that lets the scheduled polling consumer back off if there has been a number of subsequent idles or errors in a row. The multiplier is the number of polls that will be skipped before the next actual attempt happens again. When this option is in use, backoffIdleThreshold or backoffErrorThreshold must also be configured.

delay

long

500

The number of milliseconds before the next poll.

greedy

boolean

false

When true, indicates that the ScheduledPollConsumer runs immediately again if the previous run polled one or more messages.

initialDelay

long

1000

The number of milliseconds before the first poll starts.

repeatCount

long

0

The maximum limit of the number of fires. If you set this parameter to 1, the scheduler will only fire once. If you set it to 5, it will only fire five times. A value of zero or negative means the scheduler fires forever.

startScheduler

boolean

true

When true, indicates that the scheduler is automatically started.

useFixedDelay

boolean

true

When true, indicates that fixed delay is used. When false, indicates that fixed rate is used. See ScheduledExecutorService in JDK for details.

Connecting to Multiple Systems

To connect to multiple systems of a same type, use the multiple binder syntax.

For example:

spring:
  cloud:
    stream:
      binders:
      
      # 1st solace binder in this example
      solace1:
        type: solace
        environment:
          solace:
            java:
              host: tcp://localhost:55555

      # 2nd solace binder in this example
      solace2:
        type: solace
        environment:
          solace:
            java:
              host: tcp://other-host:55555

        # The only google-cloud binder
        google-cloud1:
          type: gcs
          # Add `environment` property map here if you need to customize this binder.
          # But for this example, we'll assume that defaults are used.

        # Required for internal use
        undefined:
          type: undefined
      bindings:
        input-0:
          destination: <input-destination>
          binder: google-cloud1
        output-0:
          destination: <output-destination>
          binder: solace1 # Reference 1st solace binder
        input-1:
          destination: <input-destination>
          binder: google-cloud1
        output-1:
          destination: <output-destination>
          binder: solace2 # Reference 2nd solace binder

The configuration above defines two binders of type solace and one binder of type gcs, which are then referenced within the bindings.

Each binder above is configured independently under spring.cloud.stream.binders.<bindername>.environment..

  • When connecting to multiple systems, all binder configuration must be specified using the multiple binder syntax for all binders. For example, under the spring.cloud.stream.binders.<binder-name>.environment.

  • Do not use single-binder configuration (for example, solace.java.* at the root of your application.yml) while using the multiple binder syntax.