Connector Configuration

Providing Configuration

For information about about how the connector detects configuration properties, see Spring Boot: Externalized Configuration.

Converting Canonical Spring Property Names to Environment Variables

For information about converting the Spring property names to environment variables, see the Spring documentation.

Spring Profiles

If multiple configuration files exist within the same configuration directory for use in different environments (development, production, etc.), use Spring profiles.

Using Spring profiles allow you to define different application property files under the same directory using the application-{profile}.yml filename format.

For example:

  • application.yml: The properties in non-specific files that always apply. Its properties are overridden by the properties defined in profile-specific files.
  • application-dev.yml: Defines properties specific to the development environment.
  • application-prod.yml: Defines properties specific to the production environment.

Individual profiles can then be enabled by setting the spring.profiles.active property. See Spring Boot: Profile-Specific Files for more information and an example.

Configuring Locations to Find Spring Property Files

By default, the connector detects any Spring property files that are located in Spring Boot’s default locations.

  • If you want to configure additional locations, add --spring.config.additional-location=file:<custom-config-dir>.

  • If you want to exclusively use the locations that you’ve defined and ignore Spring Boot’s default locations, add --spring.config.location=optional:classpath:/,optional:classpath:/config/,file:<custom-config-dir>.

For an example of the command to add additional locations, see Quick Start: Running the Connector Using the Command Line.

For more information about configuring locations to find Spring property files, see Spring Boot documentation.

If you want configuration files for multiple different connectors within the same config directory for use in different environments (such as, development, production, etc.), we recommend that you use Spring Boot Profiles instead of child directories.

For example, set up your configuration like this :

  • config/application-prod.yml
  • config/application-dev.yml

Do not set it up like this:

  • config/prod/application.yml
  • config/dev/application.yml

Child directories are intended to be used for merging configuration from multiple sources of configuration properties. For more information and an example showing when you might want to use multiple child directories to compose your application’s configuration, see the Spring Boot documentation.

Obtaining Build Information

Build information that includes the version, build date, time, and description is enabled by default using the Spring Boot Actuator Info Endpoint. By default, a connector only shares information related to its build.

The following example shows the structure of the output data:

{
  "build": {
    "version": "<connector version>",
    "artifact": "<connector artifact>",
    "name": "<connector name>",
    "time": "<connector build time>",
    "group": "<connector group>",
    "description": "<connector description>",
    "support": "<support information>"
  }
}

If you want to exclude build data from the output of the info endpoint, set management.info.build.enabled to false.

Alternatively, if you want to disable the info endpoint entirely, you can remove info from the list of endpoints specified in management.endpoints.web.exposure.include.

Spring Configuration Options

This connector uses Spring configuration options to customize functionality that includes Spring Cloud Binder for Solace PubSub+ and other common Spring Boot libraries.

Using the Spring Cloud Binder for Solace PubSub+

An important piece for customizing the Spring configuration options for Self-Managed Connectors is the Spring Cloud Binder for Solace PubSub+. There are configuration options available that you may find useful to configure the connector such as:

  • endpointType, which is a consumer configuration option to configure the type of endpoint (queue or topic endpoint) that consumers read messages from

  • selector, which is a consumer configuration option to configure an endpoint's message selector

  • producer configuration options to publish messages within transacted sessions

  • authentication schemes to connect to a PubSub+ event broker, such as basic authentication (default) , client certificate, or OAuth 2.0

For more information about:

Using Other Spring Boot Libraries

The following are other common references to Spring Boot libraries that you can use to configure your connector that you may find useful:

SFTP Checkpoint Store Configuration Options

The Connector for SFTP stores the current progress of file processing in a checkpoint store backed by a Last Value Queue (LVQ) on a Solace event broker:

  • For an SFTP Consumer, the checkpoint store contains information about the files that have been processed and the files that are currently being processed.

  • For an SFTP Producer, the checkpoint store contains information about the number of events written to the file and the file size (for file rotation purposes).

The following table lists the configuration options for the SFTP checkpoint store.

Configuration Option Type Valid Values Default Value Description

camel-binder.checkpoint.lvqName

String

Any

None

Required.

The name of the LVQ (spool size 0) to be used for checkpointing. The queue must exist on the same event broker and Message VPN as the target queue.

If the LVQ is deleted (administratively) or a message from the LVQ is deleted or consumed by another consumer, the Connector for SFTP will not be able to resume from the last checkpoint. In addition, the LVQ should not be shared by multiple instances of the Connector for SFTP.

camel-binder.checkpoint.autoProvisionLvq

boolean

(true | false)

false

Optional.

Set to true to auto-provision the LVQ if it does not exist.

SFTP Source Configuration Options

When configured as a source, the Connector for SFTP reads delimited, JSON, and XML files from the SFTP server, splits file data based on configuration into individual events, and sends them to the PubSub+ event broker.

The configuration properties for the SFTP source are divided into two categories: common properties (SFTP connection or file filtering etc.) and file type specific properties.

Common Configuration Options for SFTP Source

This section details the SFTP source configuration properties that are for common for all file types. For more information, see the Apache Camel SFTP component documentation.

All properties must be prefixed with spring.cloud.stream.camel.bindings.<bindingName>.consumer.endpoint.query-parameters.

Example configuration:

spring:
  cloud:
    stream:
      camel:
        bindings:
          input-0:
            consumer:
              endpoint:
                query-parameters:
                  noop: true
                  delete: false
                  includeExt: csv
                  #other properties

The following table lists the SFTP source-specific properties:

Configuration Option Type Valid Values Default Value Description

username

String

None

The username to use to log in.

password

String

None

The password to use to log in, if you are using password authentication. If it is not set, the private key file must be set.

privateKeyFile

String

None

The private key file for the SFTP endpoint to use for private key verification.

privateKeyPassphrase

String

None

The private key file passphrase for the SFTP endpoint to use for private key verification.

passiveMode

boolean

(true | false)

false

Indicates whether to use passive mode connections. The default (false) is active mode connections.

knownHostsFile

String

None

The known_hosts file for the SFTP endpoint to use for host key verification.

You can create a known_hosts file using the following command:

ssh-keyscan -H <host> -P <port>  >> known_hosts.

For example:

 ssh-keyscan -H myftpserver -p 22  >> known_hosts

strictHostKeyChecking

String

(yes | no)

no

Indicates whether to use strict host key checking. The default (no) is not to use strict host key checking.

autoCreateKnownHostsFile

boolean

(true | false)

false

Indicates whether the known_hosts file should be created automatically if it does not exist. The default (false) is not to create the file automatically.

ciphers

String

None

A comma-separated list of ciphers to use, listed in order of preference. Possible cipher names are defined by JCraft Java Secure Channel (JSch). Some examples include: aes128-ctr, aes128-cbc, 3des-ctr, 3des-cbc, blowfish-cbc, aes192-cbc, and aes256-cbc.

keyExchangeProtocols

String

None

A comma-separated list of key exchange protocols to use, listed in order of preference. Possible cipher names are defined by JCraft JSch. Some examples include: diffie-hellman-group-exchange-sha1, diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, diffie-hellman-group-exchange-sha256, ecdh-sha2-nistp256, ecdh-sha2-nistp384, and ecdh-sha2-nistp521.

publicKeyAcceptedAlgorithms

String

None

A comma-separated list of accepted public key algorithms. Some examples include: ssh-dss, ssh-rsa, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, and ecdsa-sha2-nistp521.

serverHostKeys

String

None

A comma-separated list of algorithms supported for the server host key. Some examples include: ssh-dss, ssh-rsa, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, and ecdsa-sha2-nistp521.

connectTimeout

int

> 0

10000

The time in milliseconds to wait for a connection to be established.

soTimeout

int

> 0

300000

The socket timeout, in milliseconds.

maximumReconnectAttempts

int

>= 0

None

The maximum number of times to retry to establish an initial connection to the remote FTP server. Use 0 to disable this behavior.

reconnectDelay

long

> 0

1000

The delay in milliseconds to wait before retrying the connection.

serverAliveCountMax

int

> 0

1

The maximum number of keep-alive messages to send without receiving any messages back from the server. If this threshold is reached while keep-alive messages are being sent, the connection is closed.

serverAliveInterval

int

>= 0

0

The interval (in milliseconds) between keep-alive messages. If zero is specified, no keep-alive messages are sent.

binary

boolean

(true | false)

false

The file transfer mode, either binary or ASCII. The default (false) is ASCII.

charset

String

None

The encoding of the file content.

noop

boolean

(true | false)

false

If true, the file is not deleted or moved in any way.

delete

boolean

(true | false)

false

If true, the file is deleted after it is processed successfully.

move

String

Any

None

An expression (such as Simple Language) used to dynamically set the filename when moving it after processing.

To move files into a .done subdirectory: .done

To move files into a .processed subdirectory and rename them: .processed/$#{file:name}.done

To move files into a .processed subdirectory and rename file with current timestamp: .processed/$#{date:now:yyyyMMdd}/$#{file:name.noext}-$#{date:now:yyyyMMddHHmmss}.done

moveFailed

String

Any

None

Expression (such as Simple Language) used to dynamically set the filename when moving it after processing.

To move failed files into an .error subdirectory: .error

To move failed files into an .error subdirectory and rename file: .error/$#{file:name}.error

To move failed files into an .error subdirectory and rename them with the current timestamp: .error/$#{date:now:yyyyMMdd}/$#{file:name.noext}-$#{date:now:yyyyMMddHHmmss}.error

sortBy

String

A File Language expression.

None

Built-in sort using File Language. Supports nested sorts: for example, you can sort by filename and then by modified date.

Sort by filename: $#{file:name}

Sort by file modified time: $#{file:modified}

recursive

boolean

(true | false)

false

If the specified location is a directory, the connector looks for files in all the subdirectories as well.

maxDepth

int

> 0

2147483647

The maximum depth to traverse when recursively processing a directory.

minDepth

int

> 0

None

The minimum depth to start processing when recursively processing a directory. Using minDepth=1 means the base directory. Using minDepth=2 means the first subdirectory.

antExclude

String

Any

None

Ant-style filter exclusion. If both antInclude and antExclude are used, antExclude takes precedence. Multiple exclusions may be specified in comma-delimited format.

antInclude

String

Any

None

Ant-style filter inclusion. Multiple inclusions may be specified in comma-delimited format.

antFilterCaseSensitive

boolean

(true | false)

true

Indicates whether the ant* filters are case-sensitive. The default (true) is case-sensitive.

exclude

String

A valid regular expression

None

Excludes files where the filename matches the regex pattern (matching is case in-sensitive).

include

String

A valid regular expression

None

Includes files where the filename matches the regex pattern (matching is case-insensitive).

includeExt

String

A file extension

None

Is used to include files matching file extension name (case-insensitive).

For example to include .csv files, use includeExt=csv. Multiple extensions can be separated by comma; for example to include .csv and .dat files, use includeExt=csv,dat.

excludeExt

String

A file extension

None

Is used to exclude files matching file extension name (case-insensitive).

For example to exclude .bak files, use excludeExt=bak. Multiple extensions can be separated by comma; for example to exclude .bak and .dat files, use excludeExt=bak,dat.

filterFile, filterDirectory

String

Any

None

The source SFTP server URI and the directory path.

delay

long

Any

5000

The amount of time that must elapse before the next poll.

initialDelay

long

Any

1000

The amount of time that must elapse before the first poll starts.

timeUnit

String

Enum values: NANOSECONDS, MICROSECONDS, MILLISECONDS, SECONDS, MINUTES, HOURS, DAYS

MILLISECONDS

The time unit for the initialDelay and delay options.

stopOnException

boolean

(true | false)

false

Indicates whether to stop processing the current file if an exception is encountered.

When this option is set to false (the default), the connector logs an exception and continues processing the next event in the current file.

When this option is set to true, the connector stops processing the current file and continues to the next file if any. If moveFailed is set, the file is moved to the error directory.

serverMessageLoggingLevel

String

Enum values: TRACE, DEBUG, INFO, WARN, ERROR, OFF

DEBUG

Logging levels control how much information is saved to the log file. During troubleshooting, it may be useful to raise the logging level to increase the amount of information the logs receive from the FTP server.

File Type Specific Properties

The following sections describe the properties that are specific to each file type.

All properties must be prefixed with spring.cloud.stream.camel.bindings.<bindingName>.consumer.endpoint.query-parameters.

Configuration Option Type Valid Values Default Value Description

fileType

String

Enum values: delimited, json, xml

delimited

Required. The file processor to be used the files. Default is delimited.

Delimited File Configuration Properties

The spring.cloud.stream.camel.bindings.<bindingName>.consumer.endpoint.query-parameters.fileType should be set to delimited.

Configuraton Option Type Valid Values Default Value Description

eventDelimiter

String

\n (newline)

The events in the files are separated by this character.

ignoreHeaderLine

boolean

(true | false)

false

Indicates whether the first line of the file should be ignored as a header.

  • When set to false, the first line of the file is processed as a data line.

  • When set to true, the first line of the file is not processed as a data line. Also, if set to true and paramNames is not set, the first line of the file is used as the paramNames option to convert the data to json.

paramDelimiter

String

, (comma)

The delimiter to use to separate the parameters in the header line or paramNames configuration property.

paramNames

String

None

The list of parameter names separated by the paramDelimiter. The order of the names should match the order of the parameters in the file.

If not set, and ignoreHeaderLine is true, first line of the file is used as the paramNames.

payloadFormat

String

Enum Values: json, text

json

The format of the payload.

If json, the delimited data is converted to JSON.

If text, the delimited data is kept as is.

JSON File Configuration Properties

The spring.cloud.stream.camel.bindings.<bindingName>.consumer.endpoint.query-parameters.fileType option should be set to json.

Configuration Option Type Valid Values Default Value Description

jsonPath

String

A valid JSON path expression

$

(root node)

The JSON path expression to extract, split, or filter the data from the JSON file.

The default value of $ means the entire JSON file content will be processed as a single event. For a JSON file larger than 30 MB, this value results in an exception because the maximum size of a Solace message is 30 MB. We recommend splitting the JSON file content into multiple events by configuring the JSON path expression.

XML File Configuration Properties

The spring.cloud.stream.camel.bindings.<bindingName>.consumer.endpoint.query-parameters.fileType should be set to xml.

Configuration Option Type Valid Values Default Value Description

xPath

String

a valid XPath path expression

/

(root node)

The XPath expression to extract, split, or filter the data from the XML file.

The default value of / means the entire XML file content will be processed as a single event. For XML file larger than 30MB, it would result in exception as max Solace message size is 30MB. It is recommended to split the XML file content into multiple events by configuring the XPath expression.

For example, for an employees.xml file with following content, the XPath expression to extract employee nodes would be //employee:

<employees>
    <employee>
        <name>John</name>
        <age>30</age>
    </employee>
    <employee>
        <name>Smith</name>
        <age>40</age>
    </employee>
</employees>
                        

SFTP Target Configuration Options

The following table lists the SFTP target-specific properties. For more information, see the Apache Camel SFTP Component documentation.

All properties must be prefixed with spring.cloud.stream.camel.bindings.<output-x>.producer.endpoint.query-parameters.

Example configuration:

spring:
  cloud:
    stream:
      camel:
        bindings:
          output-0:
            producer:
              endpoint:
                query-parameters:
                  username: test
                  password: test
                  fileName: output.csv
                  #other properties
Configuration Option Type Valid Values Default Value Description

username

String

None

The username to use to log in.

password

String

None

The password to use for login, if you are using password authentication. If you don't specify a password, you must use a private key file.

privateKeyFile

String

None

The private key file for the SFTP endpoint to use for private key verification.

privateKeyPassphrase

String

None

The private key file passphrase for the SFTP endpoint to use for private key verification.

passiveMode

boolean

(true | false)

false

Indicates whether to use passive mode connections. The default (false) is active mode connections.

knownHostsFile

String

None

The known_hosts file for the SFTP endpoint to use for host key verification.

You can create a known_hosts file using the following command:

ssh-keyscan -H <host> -P <port>  >> known_hosts.

For example:

 ssh-keyscan -H myftpserver -p 22  >> known_hosts

strictHostKeyChecking

String

(yes | no)

no

Indicates whether to use strict host key checking. The default (no) is not to use strict host key checking.

autoCreateKnownHostsFile

boolean

(true | false)

false

Indicates whether the known_hosts file should be created automatically if it does not exist. The default (false) is not to create the file automatically.

ciphers

String

None

A comma-separated list of ciphers to use, listed in order of preference. Possible cipher names are defined by JCraft Java Secure Channel (JSch). Some examples include aes128-ctr, aes128-cbc, 3des-ctr, 3des-cbc, blowfish-cbc, aes192-cbc, and aes256-cbc.

keyExchangeProtocols

String

None

A comma-separated list of key exchange protocols to use, listed in order of preference. Possible cipher names are defined by JCraft JSch. Some examples include: diffie-hellman-group-exchange-sha1, diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, diffie-hellman-group-exchange-sha256, ecdh-sha2-nistp256, ecdh-sha2-nistp384, and ecdh-sha2-nistp521.

publicKeyAcceptedAlgorithms

String

None

A comma-separated list of accepted public key algorithms. Some examples include ssh-dss, ssh-rsa, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, and ecdsa-sha2-nistp521.

serverHostKeys

String

None

A comma-separated list of algorithms supported for the server host key. Some examples include: ssh-dss, ssh-rsa, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, and ecdsa-sha2-nistp521.

connectTimeout

int

> 0

10000

The time in milliseconds to wait for a connection to be established.

soTimeout

int

> 0

300000

The socket timeout, in milliseconds.

maximumReconnectAttempts

int

>= 0

None

The maximum number of times to retry to establish an initial connection to the remote FTP server. Use 0 to disable this behavior.

reconnectDelay

long

> 0

1000

The delay in milliseconds to wait before retrying the connection.

serverAliveCountMax

int

> 0

1

The maximum number of keep-alive messages to send without receiving any messages back from the server. If this threshold is reached while keep-alive messages are being sent, the connection is closed.

serverAliveInterval

int

>= 0

0

The interval (in milliseconds) between keep-alive messages. If zero is specified, no keep-alive messages are sent.

binary

boolean

(true | false)

false

The file transfer mode, either binary or ASCII. The default (false) is ASCII.

charset

String

None

The character encoding of the file contents, for example utf-8.

fileExist

String

(Override | Append | Fail)

Override

The action to take if a file with the same name already exists:

  • Override: replaces the existing file.

  • Append: adds content to the existing file.

  • Fail: throws an exception indicating that there is already an existing file.

fileName

String

Any valid filename.

None

The filename to write to. It can include the leading file path. For example:

 sub1/sub2/employee.csv

If the filename is not set, a randomly generated UUID is used, in which case each event is written to a unique file.

The fileName can be specified using a Camel Simple Language expression, for example:

fileName: "backup/$#{date:now:yyyyMMdd}/test.txt"

chmod

String

Unix chmod value, for example, 740

None

The permissions (mode) to set on the file.

chmodDirectory

String

Unix chmod value, for example, 740

None

The permissions (mode) to set on the directory.

preventDuplicates

boolean

(true | false)

false

Indicates whether to prevent duplicate events from being written to the target file. Duplicate events are identified based on the value of the uniqueIdentifierHeader option. The default (false) is not to prevent duplicates.

preventDuplicatesOnlyForRedeliveredEvents

boolean

(true | false)

true

Indicates whether to check for duplicates only on redelivery. The default (true) is to check for duplicates for redelivered events only. If false, duplicate checking is performed for all events.

Use this option only when preventDuplicates is set to true.

uniqueIdentifierHeader

String

Any valid header name

eventId

The user property or header name to be used for uniquely identifying events.

Use this option only when preventDuplicates is set to true.

cacheSize

int

> 0

260

The size of the in-memory cache used to store event unique identifiers.

Use this option only when preventDuplicates is set to true.

maxEventsPerFile

Long

>= 0

0 (unlimited)

The maximum number of events to be written to a file. If the value is 0, then there is no limit.

maxFileSize

Long

>= 0

0 (unlimited)

The maximum size of the file in bytes. If the value is 0, then there is no limit.

prependToEvent

String

Any valid string

Empty string

The string to be prepended to the event.

appendToEvent

String

Any valid string

\n (newline)

The string to be appended to the event.

prependToFile

String

Any valid string

Empty string

The string to be prepended to the file (for example, a header line). This is done only once when the file is created.

prependToFirstEvent

String

Any valid string

Empty string

The string to be prepended to the first event.

serverMessageLoggingLevel

String

Enum values: TRACE, DEBUG, INFO, WARN, ERROR, OFF

DEBUG

Logging levels control how much information is saved to the log file. During troubleshooting, it may be useful to raise the logging level to increase the amount of information the logs receive from the FTP server.

Connector Configuration Options

The following table lists the configuration options. The following options in Config Option are prefixed with solace.connector.:

Config Option Type Valid Values Default Value Description

management.leader-
election.fail-over.max-attempts

int

> 0

3

The maximum number of attempts to perform a fail-over.

management.leader-
election.fail-over.back-off-initial-interval

long

> 0

1000

The initial interval (milliseconds) to back-off when retrying a fail-over.

management.leader-
election.fail-over.back-off-max-interval

long

> 0

10000

The maximum interval (milliseconds) to back-off when retrying a fail-over.

management.leader-election.fail-over.back-off-multiplier

double

>= 1.0

2.0

The multiplier to apply to the back-off interval between each retry of a fail-over.

management.leader-election.mode

enum

(standalone | active_active | active_standby)

standalone

The connector’s leader election mode.

standalone: A single instance of a connector without any leader election capabilities.

active_active: A participant in a cluster of connector instances where all instances are active.

active_standby: A participant in a cluster of connector instances where only one instance is active (i.e. the leader), and the others are standby.

management.queue

String

Any

null

The management queue name.

management.session.*

See Spring Boot Auto-Configuration for the Solace Java API for more information.

Defines the management session. This has the same interface as that used by solace.java.*.

security.enabled

boolean

(true | false)

true

If true, security is enabled. Otherwise, anyone has access to the connector’s endpoints.

security.users[<index>].name

String

Any

null

The name of the user.

security.users[<index>].password

String

Any

null

The password for the user.

security.users[<index>].roles

list<String>

admin

Empty list (i.e., read-only)

The list of roles that the specified user has. It has read-only access if no roles are returned.

Workflow Configuration Options

These configuration options are defined under the following prefixes:

  • solace.connector.workflows.<workflow-id>.: If the options support per-workflow configuration and the default prefixes.
  • solace.connector.default.workflow.: If the options support default workflow configuration.
Config Option Applicable Scopes Type Valid Values Default Value Description

enabled

Per-Workflow

boolean

(true | false)

false

If true, the workflow is enabled.

transform-headers.expressions

Per-Workflow Default

Map<String, String>

Key: A header name.

Value: A SpEL string that accepts headers as parameters.

Empty map

A mapping of header names to header value SpEL expressions.

The SpEL context contains the headers parameter that can be used to read the input message’s headers.

acknowledgment.publish-async

Per-Workflow Default

boolean

(true | false)

false

If true, publisher acknowledgment processing is done asynchronously.

The workflow’s consumer and producer bindings must support this mode, otherwise the publisher acknowledgments are processed synchronously regardless of this setting.

acknowledgment.back-pressure-threshold

Per-Workflow Default

int

>= 1

255

The maximum number of outstanding messages with unresolved acknowledgments.

Message consumption is paused when the threshold is reached to allow for producer acknowledgments to catch up.

acknowledgment.publish-timeout

Per-Workflow Default

int

>= -1

600000

The maximum amount of time (in milliseconds) to wait for asynchronous publisher acknowledgments before considering a message as failed. A value of -1 means to wait indefinitely for publisher acknowledgments.