SFTP

You can configure SFTP as a source or a target. The parameters are described in the following tables:

For message headers, see SFTP Message Headers.

SFTP Source Parameters

The following table describes the parameters for configuring SFTP as a source.

Field Description
Checkpoint Store Configuration

Checkpoint Queue

The name of the Last Value Queue (LVQ) (spool size 0) to be used for checkpointing. The queue must exist on the same event broker and Message VPN as the target queue.

The SFTP Micro-Integration stores the current progress of file processing in a checkpoint store backed by the specified LVQ.

If the LVQ is deleted (administratively) or a message from the LVQ is deleted or consumed by another consumer, the SFTPMicro-Integration will not be able to resume from the last checkpoint. In addition, the LVQ should not be shared by multiple SFTPMicro-Integrations.

Auto Create Checkpoint Queue

Select this option to automatically create the LVQ if it does not exist.
SFTP Server Configuration

SFTP Server URI

The source SFTP server URI and start directory path.

Required format: sftp://<host>:<port>/<startDirectoryName>/

For example: sftp://localhost:22/orders-data/

Username

The username to use to log in to the SFTP server.

Password

The password to use to log in to the SFTP server, if you are using password authentication. If you don't set a password, you must upload a private key file.

Private Key File

The private key file to use for private key verification.

Private Key Passphrase

The private key file passphrase to use for private key verification.

Known Hosts File

The Known Hosts file for the SFTP server to use for host key verification.

File Format

The format of the files coming from the SFTP server:

Additional Properties

(Optional) Any additional configuration related to SFTP. Click Add Context if you need to specify additional context properties. For more information, see Additional Source Parameters for SFTP

Delimited File Configuration Properties

Field Description

Event Delimiter

The events in the files are separated by this character. For example: \n (a newline character).

Field Delimiter

The fields within the events are separated by this character. For example: , (a comma).

JSON File Configuration Properties

Field Description

JSON Path Filter Expression

The JSON path expression to extract, split, or filter the data from the JSON file.

The default value of $ means the entire JSON file content will be processed as a single event. For a JSON file larger than 30 MB, this will cause an error because the maximum size of a Solace message is 30 MB. We recommend splitting the JSON file content into multiple events by configuring the JSON path expression.

XML File Configuration Properties

Field Description

XPath Filter Expression

The XPath expression to extract, split, or filter the data from the XML file.

The default value of / means the entire XML file content will be processed as a single event. For XML file larger than 30MB, it would result in exception as max Solace message size is 30MB. We recommend splitting the XML file content into multiple events by configuring the XPath expression.

For example, for an employees.xml file with following content, the XPath expression to extract employee nodes would be //employee:

<employees>
    <employee>
        <name>John</name>
        <age>30</age>
    </employee>
    <employee>
        <name>Smith</name>
        <age>40</age>
    </employee>
</employees>

Additional Source Parameters for SFTP

The following table lists the additional source-specific parameters for SFTP. For more information, see the Apache Camel SFTP Component documentation.

Configuration Option Valid Values Description

passiveMode

(true | false)

Indicates whether to use passive mode connections. The default (false) is active mode connections.

strictHostKeyChecking

(yes | no)

Indicates whether to use strict host key checking. The default (no) is not to use strict host key checking.

ciphers

A comma-separated list of ciphers to use, listed in order of preference. Possible cipher names are defined by JCraft Java Secure Channel (JSch). Some examples include: aes128-ctr, aes128-cbc, 3des-ctr, 3des-cbc, blowfish-cbc, aes192-cbc, and aes256-cbc.

keyExchangeProtocols

A comma-separated list of key exchange protocols to use, listed in order of preference. Possible cipher names are defined by JCraft JSch. Some examples include: diffie-hellman-group-exchange-sha1, diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, diffie-hellman-group-exchange-sha256, ecdh-sha2-nistp256, ecdh-sha2-nistp384, and ecdh-sha2-nistp521.

publicKeyAcceptedAlgorithms

A comma-separated list of accepted public key algorithms. Some examples include: ssh-dss, ssh-rsa, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, and ecdsa-sha2-nistp521.

serverHostKeys

A comma-separated list of algorithms supported for the server host key. Some examples include: ssh-dss, ssh-rsa, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, and ecdsa-sha2-nistp521.

connectTimeout

> 0

The time in milliseconds to wait for a connection to be established.

soTimeout

> 0

The socket timeout, in milliseconds.

maximumReconnectAttempts

>= 0

The maximum number of times to retry to establish an initial connection to the remote FTP server. Use 0 to disable this behavior.

reconnectDelay

> 0

The delay in milliseconds to wait before retrying the connection.

serverAliveCountMax

> 0

The maximum number of keep-alive messages to send without receiving any messages back from the server. If this threshold is reached while keep-alive messages are being sent, the connection is closed.

serverAliveInterval

>= 0

The interval (in milliseconds) between keep-alive messages. If zero is specified, no keep-alive messages are sent.

binary

(true | false)

The file transfer mode, either binary or ASCII. The default (false) is ASCII.

charset

A valid charset value.

The character encoding of the file contents, for example utf-8.

noop

(true | false)

If true, the file is not deleted or moved in any way.

delete

(true | false)

If true, the file is deleted after it is processed successfully.

move

Any

An expression (such as Simple Language) used to dynamically set the filename when moving it after processing.

To move files into a .done subdirectory: .done

To move files into a .processed subdirectory and rename them: .processed/$#{file:name}.done

To move files into a .processed subdirectory and rename file with current timestamp: .processed/$#{date:now:yyyyMMdd}/$#{file:name.noext}-$#{date:now:yyyyMMddHHmmss}.done

moveFailed

Any

Expression (such as Simple Language) used to dynamically set the filename when moving it after processing.

To move failed files into an .error subdirectory: .error

To move failed files into an .error subdirectory and rename file: .error/$#{file:name}.error

To move failed files into an .error subdirectory and rename them with the current timestamp: .error/$#{date:now:yyyyMMdd}/$#{file:name.noext}-$#{date:now:yyyyMMddHHmmss}.error

sortBy

A File Language expression.

Built-in sort using File Language. Supports nested sorts: for example, you can sort by filename and then by modified date.

Sort by filename: $#{file:name}

Sort by file modified time: $#{file:modified}

recursive

(true | false)

If the specified location is a directory, the connector looks for files in all the subdirectories as well.

maxDepth

> 0

The maximum depth to traverse when recursively processing a directory.

minDepth

> 0

The minimum depth to start processing when recursively processing a directory. Using minDepth=1 means the base directory. Using minDepth=2 means the first subdirectory.

antExclude

Any

Ant-style filter exclusion. If both antInclude and antExclude are used, antExclude takes precedence. Multiple exclusions may be specified in comma-delimited format.

antInclude

Any

Ant-style filter inclusion. Multiple inclusions may be specified in comma-delimited format.

antFilterCaseSensitive

(true | false)

Indicates whether the ant* filters are case-sensitive. The default (true) is case-sensitive.

exclude

A valid regular expression

Excludes files where the filename matches the regex pattern (matching is case in-sensitive).

include

A valid regular expression

Includes files where the filename matches the regex pattern (matching is case-insensitive).

includeExt

A file extension

Is used to include files matching file extension name (case-insensitive).

For example to include .csv files, use includeExt=csv. Multiple extensions can be separated by comma; for example to include .csv and .dat files, use includeExt=csv,dat.

excludeExt

A file extension

Is used to exclude files matching file extension name (case-insensitive).

For example to exclude .bak files, use excludeExt=bak. Multiple extensions can be separated by comma; for example to exclude .bak and .dat files, use excludeExt=bak,dat.

filterFile, filterDirectory

Any

The source SFTP server URI and the directory path.

delay

Any

The amount of time that must elapse before the next poll.

initialDelay

Any

The amount of time that must elapse before the first poll starts.

timeUnit

Enum values: NANOSECONDS, MICROSECONDS, MILLISECONDS, SECONDS, MINUTES, HOURS, DAYS

The time unit for the initialDelay and delay options.

stopOnException

(true | false)

Indicates whether to stop processing the current file if an exception is encountered.

When this option is set to false (the default), the connector logs an exception and continues processing the next event in the current file.

When this option is set to true, the connector stops processing the current file and continues to the next file if any. If moveFailed is set, the file is moved to the error directory.

serverMessageLoggingLevel

Enum values: TRACE, DEBUG, INFO, WARN, ERROR, OFF

Logging levels control how much information is saved to the log file. During troubleshooting, it may be useful to raise the logging level to increase the amount of information the logs receive from the FTP server.

maxFileSize

>= 0

The maximum size, in bytes, of the files to be consumed. This option helps to prevent Out Of Memory (OOM) errors on systems with a small amount of available memory.

SFTP Target Parameters

The following table describes the parameters for configuring SFTP as a target.

Configuration Option Description
Checkpoint Store Configuration

Checkpoint Queue

The name of the Last Value Queue (LVQ) (spool size 0) to be used for checkpointing. The queue must exist on the same event broker and Message VPN as the target queue.

The Micro-Integration for SFTP stores the current progress of file processing in a checkpoint store backed by the specified LVQ.

If the LVQ is deleted (administratively) or a message from the LVQ is deleted or consumed by another consumer, the Micro-Integration for SFTP will not be able to resume from the last checkpoint. In addition, the LVQ should not be shared by multiple instances of the Micro-Integration for SFTP.

Auto Create Checkpoint Queue

Select this option to automatically create the LVQ if it does not exist.
SFTP Server Configuration

SFTP Server URI

The target SFTP server URI and start directory path.

Required format: sftp://<host>:<port>/<startDirectoryName>/

For example: sftp://localhost:22/orders-data/

Username

The username to use to log in to the SFTP server.

Password

The password to use to log in to the SFTP server, if you are using password authentication. If you don't set a password, you must upload a private key file.

Private Key File

The private key file to use for private key verification.

Private Key Passphrase

The private key file passphrase to use for private key verification.

Known Hosts File

The Known Hosts file for the SFTP server to use for host key verification.

Write Mode

The action to take if a file with the same name already exists:

  • Append—Events are added to the existing file.

  • Override—The existing file is replaced.

  • Fail—An error is generated, indicating that there is already an existing file.

Maximum File Size

The maximum size of the file in bytes. If the value is 0, then there is no limit.

Maximum Number of Events

The maximum number of events to be written to a file, after which a new file is created. If the value is 0, then there is no limit.

Additional Properties

(Optional) Any additional configuration related to SFTP. Click Add Context if you need to specify additional context properties. For more information, see Additional Target Parameters for SFTP.

Additional Target Parameters for SFTP

The following table lists the additional target-specific parameters for SFTP. For more information, see the Apache Camel SFTP Component documentation.

Configuration Option Valid Values Description

passiveMode

(true | false)

Indicates whether to use passive mode connections. The default (false) is active mode connections.

strictHostKeyChecking

(yes | no)

Indicates whether to use strict host key checking. The default (no) is not to use strict host key checking.

ciphers

A comma-separated list of ciphers to use, listed in order of preference. Possible cipher names are defined by JCraft Java Secure Channel (JSch). Some examples include aes128-ctr, aes128-cbc, 3des-ctr, 3des-cbc, blowfish-cbc, aes192-cbc, and aes256-cbc.

keyExchangeProtocols

A comma-separated list of key exchange protocols to use, listed in order of preference. Possible cipher names are defined by JCraft JSch. Some examples include: diffie-hellman-group-exchange-sha1, diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, diffie-hellman-group-exchange-sha256, ecdh-sha2-nistp256, ecdh-sha2-nistp384, and ecdh-sha2-nistp521.

publicKeyAcceptedAlgorithms

A comma-separated list of accepted public key algorithms. Some examples include ssh-dss, ssh-rsa, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, and ecdsa-sha2-nistp521.

serverHostKeys

A comma-separated list of algorithms supported for the server host key. Some examples include: ssh-dss, ssh-rsa, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, and ecdsa-sha2-nistp521.

connectTimeout

> 0

The time in milliseconds to wait for a connection to be established.

soTimeout

> 0

The socket timeout, in milliseconds.

maximumReconnectAttempts

>= 0

The maximum number of times to retry to establish an initial connection to the remote FTP server. Use 0 to disable this behavior.

reconnectDelay

> 0

The delay in milliseconds to wait before retrying the connection.

serverAliveCountMax

> 0

The maximum number of keep-alive messages to send without receiving any messages back from the server. If this threshold is reached while keep-alive messages are being sent, the connection is closed.

serverAliveInterval

>= 0

The interval (in milliseconds) between keep-alive messages. If zero is specified, no keep-alive messages are sent.

binary

(true | false)

The file transfer mode, either binary or ASCII. The default (false) is ASCII.

charset

A valid charset value.

The character encoding of the file contents, for example utf-8.

fileExist

(Override | Append | Fail)

The action to take if a file with the same name already exists:

  • Override: replaces the existing file.

  • Append: adds content to the existing file.

  • Fail: throws an exception indicating that there is already an existing file.

fileName

Any valid filename.

The filename to write to. It can include the leading file path. For example:

 sub1/sub2/employee.csv

If the filename is not set, a randomly generated UUID is used, in which case each event is written to a unique file.

The fileName can be specified using a Camel Simple Language expression, for example:

fileName: "backup/$#{date:now:yyyyMMdd}/test.txt"

chmod

Unix chmod value, for example, 740

The permissions (mode) to set on the file.

chmodDirectory

Unix chmod value, for example, 740

The permissions (mode) to set on the directory.

preventDuplicates

(true | false)

Indicates whether to prevent duplicate events from being written to the target file. Duplicate events are identified based on the value of the uniqueIdentifierHeader option. The default (false) is not to prevent duplicates.

preventDuplicatesOnlyForRedeliveredEvents

(true | false)

Indicates whether to check for duplicates only on redelivery. The default (true) is to check for duplicates for redelivered events only. If false, duplicate checking is performed for all events.

Use this option only when preventDuplicates is set to true.

uniqueIdentifierHeader

Any valid header name

The user property or header name to be used for uniquely identifying events.

Use this option only when preventDuplicates is set to true.

cacheSize

> 0

The size of the in-memory cache used to store event unique identifiers.

Use this option only when preventDuplicates is set to true.

maxEventsPerFile

>= 0

The maximum number of events to be written to a file. If the value is 0, then there is no limit.

maxFileSize

>= 0

The maximum size of the file in bytes. If the value is 0, then there is no limit.

prependToEvent

Any valid string

The string to be prepended to the event.

appendToEvent

Any valid string

The string to be appended to the event.

prependToFile

Any valid string

The string to be prepended to the file (for example, a header line). This is done only once when the file is created.

prependToFirstEvent

Any valid string

The string to be prepended to the first event.

serverMessageLoggingLevel

Enum values: TRACE, DEBUG, INFO, WARN, ERROR, OFF

Logging levels control how much information is saved to the log file. During troubleshooting, it may be useful to raise the logging level to increase the amount of information the logs receive from the FTP server.