Substitution Expressions Overview

Substitution expressions are a Solace-specific expression language used to replace specific text attributes (request targets, request headers, etc.) with system generated output.

For example, suppose your deployment warrants using REST Delivery Points (RDPs) to send messages into storage services such as Azure Storage, Amazon S3, or Google Cloud Storage. This requires sending each message to a unique target, preferably with that uniqueness coming from the message topic, metadata, or the current time. To enable this, you could embed a substitution expression into the request target that configures the system to modify the target based on the data (topic, current time, etc.) from each incoming message.

A substitution expression consists of either a literal or a function call. Function call parameters can be any substitution expression that is of the appropriate type. When embedded in text attributes, substitution expressions are delimited by ${ and }. Only close braces outside of string literals end a substitution expression; there is no special escaping of braces required within a substitution expression. To include ${ in the text, a substitution expression can be used. For example, the ${ can be encoded as ${"$"}{.

Any substitution expression that isn't valid results in an error at configure time. You can view error message details through all available monitoring interfaces (Broker Manager, CLI, SEMPv1, and SEMPv2).

Substitution Expression Functions

Substitution expression functions return a value and possibly take parameters. The currently available functions and whether they are applicable to RDP, Kafka sender, or Kafka receiver objects, are listed in the following table.

Function parameters and return values have specified types. A string may be passed in anywhere bytes are accepted; there are no other automatic conversions. The available types are:

  • integer: a 64-bit integer value (range: -9223372036854775808 through 9223372036854775807).

  • string: a UTF-8 encoded string.

  • bytes: a sequence of 0 or more bytes.

The integer timestamp parameter passed to all the time functions (utcYear, utcDate etc.) must be in milliseconds.

Function Name Usage Parameter Types Return Type Context Description
RDP Kafka  
Sender Receiver  

topic

topic([N])

integer

string

Applicable Applicable Not applicable

Returns part or all of the Solace message destination topic. If N is positive, returns the Nth field, numbering from 1. If N is negative, returns the Nth field from the end of the topic. If N is 0 or not specified, returns the entire topic string. If the selected field doesn't exist, returns the empty string. For example, if the topic was alfa/bravo/charlie/delta, then topic(2) would return bravotopic(-2) would return charlie, topic() or topic(0) would return alfa/bravo/charlie/delta, and topic(8) or topic(-7) would return the empty string.

kafkaTopic

kafkaTopic([N, separator])

 

integer, string

string

Not applicable Not applicable Applicable

Returns the topic from a Kafka message.

If N is provided, separator will be used as the topic level separator. The separator is required as Kafka does not define this (although by convention dot and underscore are common).

 

now

now()

N/A

integer

Applicable Applicable Applicable

Returns the current time in milliseconds from midnight, January 1, 1970 UTC. For example, if the time were Mon 22 Jun 2020 13:13:02.123 EDT, the value returned would be 1592845982123. Note that due to implementation limitations, the accuracy and resolution of the milliseconds may be low. It is guaranteed that the now() value will never decrease unless the clock on the event broker is updated or corrected.

senderTimestamp

senderTimestamp()

N/A

integer

Applicable Applicable Not applicable

Returns the senderTimestamp property of the message in milliseconds from midnight, January 1, 1970 UTC, or 0 if the senderTimestamp property is not set (which means a timestamp of midnight, January 1, 1970 UTC).1

kafkaTimestamp

kafkaTimestamp()

 

integer

Not applicable Not applicable Applicable

Returns the timestamp message field from the Kafka message.

ts

ts()

N/A

integer

Applicable Applicable Applicable

If the senderTimestamp property is present on the message, the value of senderTimestamp() is returned, otherwise the value of now() is returned.

utcDate

utcDate([separator [, timestamp]])

string, integer

string

Applicable Applicable Applicable

Returns the date portion of the timestamp in UTC, formatted as YYYY-MM-DD. If provided, separator is used instead of hyphens to separate the parts. If timestamp is omitted, the value returned by ts() is used.

utcYear

utcYear([timestamp])

integer

string

Applicable Applicable Applicable

Returns the four digit UTC year. If timestamp is omitted, the value returned by ts() is used.

utcMonth

utcMonth([timestamp])

integer

string

Applicable Applicable Applicable

Returns the two digit UTC month. If timestamp is omitted, the value returned by ts() is used.

utcDay

utcDay([timestamp])

integer

string

Applicable Applicable Applicable

Returns the two digit UTC day. If timestamp is omitted, the value returned by ts() is used.

utcTime

utcTime([separator [, timestamp]])

string, integer

string

Applicable Applicable Applicable

Returns the time portion of the timestamp in UTC, formatted as hh:mm:ss. If provided, separator is used instead of colons to separate the parts. If timestamp is omitted, the value returned by ts() is used.

utcHour

utcHour([timestamp])

integer

string

Applicable Applicable Applicable

Returns the two digit UTC hour. If timestamp is omitted, the value returned by ts() is used.

utcMinute

utcMinute([timestamp])

integer

string

Applicable Applicable Applicable

Returns the two digit UTC minute. If timestamp is omitted, the value returned by ts() is used.

localDate

localDate([separator [, timestamp]])

string, integer

string

Applicable Applicable Applicable

Returns the date portion of the timestamp in the broker's local time zone, formatted as YYYY-MM-DD. If provided, separator is used instead of hyphens to separate the parts. If timestamp is omitted, the value returned by ts() is used.

localYear

localYear([timestamp])

integer

string

Applicable Applicable Applicable

Returns the four digit year portion of the specified timestamp in the broker's local time zone. If timestamp is omitted, the value returned by ts() is used.

localMonth

localMonth([timestamp])

integer

string

Applicable Applicable Applicable

Returns the two digit month portion of the specified timestamp in the broker's local time zone. If timestamp is omitted, the value returned by ts() is used.

localDay

localDay([timestamp])

integer

string

Applicable Applicable Applicable

Returns the two digit day portion of the specified timestamp in the broker's local time zone. If timestamp is omitted, the value returned by ts() is used.

localTime

localTime([separator [, timestamp]])

string, integer

string

Applicable Applicable Applicable

Returns the time portion of the timestamp in the broker's local time zone, formatted as hh:mm:ss. If provided, separator is used instead of colons to separate the parts. If timestamp is omitted, the value returned by ts() is used.

localHour

localHour([timestamp])

integer

string

Applicable Applicable Applicable

Returns the two digit hour portion of the specified timestamp in the broker's local time zone. If timestamp is omitted, the value returned by ts() is used.

localMinute

localMinute([timestamp])

integer

string

Applicable Applicable Applicable

Returns the two digit minute portion of the specified timestamp in the broker's local time zone. If timestamp is omitted, the value returned by ts() is used.

second

second([timestamp])

integer

string

Applicable Applicable Applicable

Returns the two digit second portion of the specified timestamp. If timestamp is omitted, the value returned by ts() is used.

millisecond

millisecond([timestamp])

integer

string

Applicable Applicable Applicable

Returns the three digit millisecond portion of the specified timestamp. If timestamp is omitted, the value returned by ts() is used.

unixTime

unixTime([timestamp])

integer

integer

Applicable Applicable Applicable

Returns the number of seconds since midnight, January 1, 1970 UTC for the specified timestamp, ignoring leap seconds.  This is known as unix time or epoch time.  For example, if the provided timestamp was for Mon 22 Jun 2020 13:13:02.123 EDT, the value returned would be 1592845982. If timestamp is omitted, the value returned by ts() is used.

randomBytes

randomBytes(N)

integer

bytes

Applicable Applicable Applicable

Returns N cryptographically random bytes.  N must be a positive integer less than 100.

uuid

UUID

uuid()

UUID()

N/A

string

Applicable Applicable Applicable

Returns a new UUID. The uuid() function generates lowercase letters; the UUID() function generates uppercase letters.  For example, UUID() might return the string D64DB3F0-63A8-4D9B-B605-FC30B5216552. Each time these functions are called, a different UUID is generated.

base32

BASE32

base32NoPad

BASE32NoPad

base32(data)

BASE32(data)

base32NoPad(data)

BASE32NoPad(data)

bytes

string

Applicable Applicable Applicable

Returns the provided data (which can be of type string or bytes) encoded as base32. There are four variants, for the four combinations of uppercase/lowercase and padding.  The variants that start with base32 will produce lowercase output; the variants that start with BASE32 will product uppercase output.  The variants that end with NoPad do not add padding.

base64

base64NoPad

base64(data)

base64NoPad(data)

bytes

string

Applicable Applicable Applicable

Returns the provided data (which can be of type string or bytes) encoded as base64. The base64NoPad variation omits padding.

base64Url

base64UrlNoPad

base64Url(data)

base64UrlNoPad(data)

bytes

string

Applicable Applicable Applicable

Returns the provided data (which can be of type string or bytes) encoded as base64url. The base64UrlNoPad variation omits padding.

hex

HEX

hex(data)

HEX(data)

bytes

string

Applicable Applicable Applicable

Return the provided data (which can be of type string or bytes) encoded as hexadecimal (base16).  The hex variant produces lowercase output and the HEX variant produces uppercase output.

urlEncode

urlEncode(data [, exceptions])

bytes, string

string

Applicable Applicable Applicable

Returns the provided data (which can be of type string or bytes) URL encoded:  any characters that are not unreserved characters according to RFC 3986 section 2.3 are percent-encoded (with the hexadecimal digits A-F in uppercase as per that specification), unless the character is specified in the exceptions string.  If exceptions is not provided, it defaults to the empty string.

senderId

senderId()

N/A

string

Applicable Applicable Not applicable

Returns the Solace senderId property of the message, or the empty string if the senderId property is not set.1

userPropertyAsString

userPropertyAsString(propertyName)

string

string

Applicable Applicable Not applicable

Returns the propertyName (a case-sensitive string) user property value as a string.  Returns the empty string if there is no user property with the specified name.  If there is more than one user property with the specified name, only one of them is returned (it is undefined which one).1

kafkaHeader

kafkaHeader(headerName)

string

bytes

Not applicable Not applicable Applicable

Gets a header field from the Kafka message. Behaviours for missing or duplicated headerName is the same as userPropertyAsString().

kafkaHeaderAsString

kafkaHeaderAsString(headerName)

string

string

Not applicable Not applicable Applicable

Returns the same vale as kafkaHeader(), but with bytes converted to a string. Each byte is interpreted as an ASCII character, truncated at the first non-printable character.

msgType

msgType()

N/A

string

Applicable Applicable Not applicable

Returns the Solace Application Message Type (a string).  Note that there are several different things that message type can refer to. In this context it is an arbitrary value set by the client and passed along but otherwise ignored by the broker.  In JMS it is accessed as JMSType; in CCSMP it can be set via solClient_msg_setApplicationMsgType.1

msgId

msgId()

N/A

string

Applicable Applicable Not applicable

Returns the Solace application message ID, a string, or an empty string if the property is not set. This property is not automatically generated by the event broker and must be set by the publisher application.1

sequenceNumber

sequenceNumber()

N/A

integer

Applicable Applicable Not applicable

Returns the sequence number (an integer) of the message.1

correlationId

correlationId()

N/A

string

Applicable Applicable Not applicable

Returns the Solace correlation ID (a string) of the message.1

partitionKey

partitionKey()

N/A

string

Applicable Applicable Not applicable

Returns the Solace partitioned queue key of the message.1

kafkaPartitionKey

kafkaPartitionKey()

N/A

bytes

Not applicable Not applicable Applicable

Returns the Kafka partition key of the message as bytes.

kafkaPartitionKeyAsString

kafkaPartitionKeyAsString()

N/A

string

Not applicable Not applicable Applicable

Returns the same value as kafkaPartitionKey(), but with bytes converted to a string. Each byte is interpreted as an ASCII character, truncated at the first non-printable character.

kafkaPartitionNumber

kafkaPartitionNumber()

N/A

integer

Not applicable Not applicable Applicable

Returns the partition number the message was received from.

kafkaPartitionOffset

kafkaPartitionOffset()

N/A

integer

Not applicable Not applicable Applicable

Returns the partition offset of the message.

replace

replace(source, old, new [, count])

string, string, string, integer

string

Applicable Applicable Applicable

Returns source with string old replaced with new, up to count number of times starting from the first (or last if count is negative). If count is not provided, all occurrences of old are replaced. While of general use, this has been added to convert Kafka Topics (with dots or underscores) to Solace Topics (with slashes).

withDefault

withDefault(valuedefault)

string, string

string

Applicable Applicable Applicable

Returns value unless value is the empty string, in which case it returns default.

1—These values map to specific fields in non-SMF APIs. For information on how the values returned from these substitution expressions map from SMF to non-SMF protocols, see the following sections on our Protocol Metadata and Payload Encoding Interactions page:

Substitution Expression Processing Examples

The following table lists several examples that illustrate how PubSub+ event brokers process substitution expressions. The examples assume a message arriving at January 2, 2006 3:04:05.123 pm with the following data:

  • topic=t/v1/DE/sk/escalator/1,2/sk_1.x/DE12345/raw

  • sender timestamp=January 2, 2006 3:04:04.987 pm

  • "author" user property=Jane "The Bear" Doe

  • In specific contexts in which substitution expressions are enabled, further processing may happen after substitution expression evaluation. For example, after substitution processing request targets will have spaces and non-ASCII characters percent-encoded to ensure that the target is valid.

  • Line breaks and indentation in the following examples are for readability only.

Input Output

${topic()}

t/v1/DE/sk/escalator/1,2/sk_1.x/DE12345/raw

${urlEncode(topic(), "/")}

t/v1/DE/sk/escalator/1%2C2/sk_1.x/DE12345/raw

${urlEncode(topic())}

t%2Fv1%2FDE%2Fsk%2Fescalator%2F1%2C2%2Fsk_1.x%2FDE12345%2Fraw

global/landing/solace/${topic(5)}/${topic(7)}/${topic(8)}/${ts()}

global/landing/solace/escalator/sk_1.x/DE12345/1136189044987

${utcDate()}/${utcTime()}/${base32(randomBytes(15))}

2006-01-02/15:04:04/4fdm3hlyafhh77ntxqcwbt7b

${utcDate("/")}/${utcTime("/")}/${BASE32(randomBytes(15))}

2006/01/02/15/04/04/4FDM3HLYAFHH77NTXQCWBT7B

${userPropertyAsString("author")}

Jane "The Bear" Doe

${"${"}text in the delimiters}

${text in the delimiters}

A number:  ${5}

A number:  5

${"${This is a valid substitution expression.}"}

${This is a valid substitution expression.}

${"This is also a valid substitution expression!  :-}"}

This is also a valid substitution expression!  :-}

This is a valid attribute value with no substitution expression. :-}

This is a valid attribute value with no substitution expression. :-}

${base64("Strings can be used where bytes are specified.")}

U3RyaW5ncyBjYW4gYmUgdXNlZCB3aGVyZSBieXRlcyBhcmUgc3BlY2lmaWVkLg==

Text Attributes That Can Contain Substitution Expressions

The following text attributes can contain substitution expressions: