Configuring Connection Details

This section provides instructions for configuring the connection details required to establish communication between the Micro-Integration and your third-party system.

For information about configuring the connection to the event broker, see Connecting to Your Event Broker.

This Micro-Integration supports workflows only in the direction of:

MongoDB to Solace

The name of the binder for MongoDB CDC is mongocdc.

To share settings across all workflows in a Micro-Integration, configure the parameters using default instead of a workflow-specific identifier (for example, input-0). This reduces repetition when multiple workflows use the same parameter values. Workflow-specific settings override default settings when both are present.

MongoDB Database Prerequisites

Before you use the Micro-Integration for MongoDB CDC, you must configure your MongoDB database with one of the following deployment types:

Replica Set

The Micro-Integration for MongoDB CDC can capture changes from a single MongoDB replica set. Production replica sets require a minimum of at least three members.

To use the Micro-Integration for MongoDB CDC with a replica set, you must set the value of the mongodbConnectionString property in the connector configuration to the replica set connection string. When the connector is ready to begin capturing changes from a MongoDB change stream, it starts a connection task. The connection task then uses the specified connection string to establish a connection to an available replica set member.

Because standalone servers do not have an oplog, the Micro-Integration for MongoDB CDC cannot monitor changes for standalone MongoDB servers. You must convert a standalone server to a replica set with one member.
Sharded Cluster

A MongoDB sharded cluster consists of:
- One or more shards, each deployed as a replica set
- A separate replica set that acts as the cluster's configuration server
- One or more routers (also called mongos) to which clients connect and that routes requests to the appropriate shards
To use the Micro-Integration for MongoDB CDC with a sharded cluster, in the configuration, set the value of the mongodbConnectionString property to the sharded cluster connection string.

Optimal Oplog Configuration

The Micro-Integration for MongoDB CDC reads change streams to obtain oplog data for a replica set. Because the oplog is a fixed-sized, capped collection, if it exceeds its maximum configured size, it begins to overwrite its oldest entries. If the Micro-Integration is stopped for any reason, when it restarts, it attempts to resume streaming from the last oplog stream position. However, if last stream position was removed from the oplog, depending on the value specified in the connector's snapshotMode property, the Micro-Integration might fail to start.

To ensure that the oplog retains the offset values the Micro-Integration requires to resume streaming, you can use either of the following approaches:

Increase the size of the oplog. Based on your typical workloads, set the oplog size to a value that is greater than the peak number of oplog entries per hour.
Increase the minimum number of hours that an oplog entry is retained (MongoDB 4.4 and greater). This setting is time-based, such that entries in the last n hours are guaranteed to be available even if the oplog reaches its maximum configured size. Although this is generally the preferred option, for clusters with high workloads that are nearing capacity, specify the maximum oplog size.

To help prevent failures that are related to missing oplog entries, it's important to track metrics that report replication behavior, and to optimize the oplog size to support the Micro-Integration. In particular, you should monitor the values of Oplog GB/Hour and Replication Oplog Window. If the Micro-Integration is offline for an interval that exceeds the value of the replication oplog window, and the primary oplog grows faster than the Micro-Integration can consume entries, a Micro-Integration failure can result.

For more information about how to monitor these metrics, see the MongoDB documentation.

MongoDB Connection Details

To manually configure the MongoDB connection details, set the following properties in application.yml:

spring:
  cloud:
    stream:
      mongocdc:
        bindings:
          input-0:
            consumer:
              endpoint:
                query-parameters:
                  mongodbConnectionString: <connection string>
                  mongodbUser: <username>
                  mongodbPassword: <password>
                  mongodbAuthsource: <auth database>
                  <SSL parameters>
                  ...

Where:

mongodbConnectionString is the connection string URI for MongoDB. For example, mongodb://host:port/?replicaSet=rs0.
mongodbUser is the username to use for authenticating with MongoDB.
mongodbPassword is the password to use for authenticating with the MongoDB database.
mongodbAuthsource is the authentication source database. For example, admin.
<SSL parameters> are optional SSL/TLS configuration parameters. For more information, see SSL/TLS Configuration Parameters.

Basic Authentication Parameters

Field	Type	Description
`mongodbConnectionString`	`String`	The connection string URI for MongoDB. For example, `mongodb://host:port/?replicaSet=rs0`.
`mongodbUser`	`String`	The username to use for authenticating with MongoDB.
`mongodbPassword`	`String`	The password to use for authenticating with the MongoDB database.
`mongodbAuthsource`	`String`	The authentication source database. For example, `admin`.

SSL/TLS Configuration Parameters

For SSL/TLS certificate-based authentication, configure the following SSL properties in addition to the basic connection parameters.

Field	Type	Description
`mongodbSslEnabled`	`boolean`	When `true`, indicates that an SSL connection to MongoDB is enabled.
`mongodbSslValidateHostname`	`boolean`	When `true`, indicates that the hostname is validated against the server certificate's hostname.
`mongodbSslKeystore`	`String`	The path to the keystore file containing the client certificate. For example, `path/ssl/client.p12`.
`mongodbSslKeystoreType`	`String`	The type of the SSL keystore. Valid values are `PKCS12` and `JKS`.
`mongodbSslKeystorePassword`	`String`	The password for the SSL keystore.
`mongodbSslTruststore`	`String`	The path to the truststore file containing trusted certificates. For example, `path/ssl/truststore.jks`.
`mongodbSslTruststoreType`	`String`	The type of the SSL truststore. Valid values are `PKCS12` and `JKS`.
`mongodbSslTruststorePassword`	`String`	The password for the SSL truststore.

Checkpoint Store Configuration

The Micro-Integration for MongoDB CDC stores the information about the current progress of processing change events in a checkpoint store backed by a Last Value Queue (LVQ) on a Solace event broker. The checkpoint store contains information about the oplog position that has been processed.

For example, add the following settings to the application.yml file:

solace:
  mongocdc:
    checkpoint:
      lvqName: lvq-checkpoint
      autoProvisionLvq: true

The following table lists the configuration options for the Checkpoint Store.

Config Option Type Valid Values Default Value Description

Config Option	Type	Valid Values	Default Value	Description
`solace.mongocdc.checkpoint.lvqName`	`String`		None	Required. The name of the event broker Last Value Queue (spool size 0) to be used for checkpointing. The queue must exist on the same event broker and Message VPN as the target queue. If the LVQ is deleted (administratively) or a message from the LVQ is deleted or consumed by another consumer, the Micro-Integration for MongoDB CDC will not be able to resume from the last checkpoint. In addition, the LVQ should not be shared by multiple instances of the Micro-Integration for MongoDB CDC.
`solace.mongocdc.checkpoint.autoProvisionLvq`	`boolean`	`true, false`	`false`	Optional. When `true`, indicates that the LVQ queue will be auto-provisioned if it does not exist.

solace.mongocdc.checkpoint.lvqName

String

None

Required.

The name of the event broker Last Value Queue (spool size 0) to be used for checkpointing. The queue must exist on the same event broker and Message VPN as the target queue.

If the LVQ is deleted (administratively) or a message from the LVQ is deleted or consumed by another consumer, the Micro-Integration for MongoDB CDC will not be able to resume from the last checkpoint. In addition, the LVQ should not be shared by multiple instances of the Micro-Integration for MongoDB CDC.

solace.mongocdc.checkpoint.autoProvisionLvq

boolean

true, false

false

Optional.

When true, indicates that the LVQ queue will be auto-provisioned if it does not exist.

MongoDB CDC Binder Configuration Options

The following properties are available at the MongoDB CDC binder level.

Config Option Type Valid Values Description

Config Option	Type	Valid Values	Description
`spring.cloud.stream.bindings.input-<workflow-id>.destination`	`String`	`Format:` `database.collection` For example, `inventory.products`	The MongoDB collection from which to consume change events. The destination must include the database name and the name of the collection.
`spring.cloud.stream.bindings.input-<workflow-id>.binder`	`String`	`mongocdc`	This property must be set to `mongocdc`.

spring.cloud.stream.bindings.input-<workflow-id>.destination

String

Format:

database.collection

For example, inventory.products

The MongoDB collection from which to consume change events. The destination must include the database name and the name of the collection.

spring.cloud.stream.bindings.input-<workflow-id>.binder

String

mongocdc

This property must be set to mongocdc.

MongoDB CDC Consumer Configuration Options

The following configuration options are available for the MongoDB CDC consumer.

All properties must be prefixed with spring.cloud.stream.mongocdc.bindings.input-<workflow-id>.consumer.endpoint.query-parameters.

Example configuration:

spring:
  cloud:
    stream:
      mongocdc:
        bindings:
           input-0:
             consumer:
               endpoint:
                 query-parameters:
                   <connection parameters>
                   skipDeleteEvents: false
                   snapshotMode: no_data
                   maxBatchSize: 127
                   maxQueueSize: 255

Config Option	Type	Valid Values	Default Value	Description
`skipDeleteEvents`	`boolean`	`true`, `false`	`false`	When `true`, indicates that DELETE operation events will be skipped.
`snapshotMode`	`String`	`initial`, `schema_only`, `when_needed`, `never`, `no_data`, `initial_only`	`no_data`	The criteria for running a snapshot upon startup of the connector. The `initial` (default) option runs a snapshot that captures the current full state of the configured collections if the connector does not detect any offsets for the logical server name. After the snapshot completes, the connector begins to stream changes from the oplog. The `never` or `no_data` options do not run a snapshot. Upon first startup, the Micro-Integrationimmediately begins reading from the beginning of the oplog, but it does not create READ events to represent the data set at the point of the connector's start-up.
`skippedOperations`	`String`	`c`, `u`, `d`, `t`, `none`	`t`	A comma-separated list of operation types to skip during streaming. Valid options include: `c` for inserts/create; `u` for updates; `d` for deletes, `t` for truncates, and `none` to indicate nothing skipped.
`offsetFlushIntervalMs`	`Integer`		`60000`	The interval for flushing offsets in milliseconds.
`maxQueueSize`	`Integer`		`255`	The maximum size of the queue for change events read from the database log but not yet recorded or forwarded. This value should always be larger than the maximum batch size.
`maxBatchSize`	`Integer`		`127`	The maximum size of each batch of source records.

Provide feedback