MongoDB Bulk CDC

The Micro-Integration for MongoDB Bulk CDC captures changes (inserts, updates, deletes) from multiple MongoDB database collections simultaneously (using Debezium) and streams them to topics on an event broker service. Only header mapping is supported; payload mapping is not available for this Micro-Integration.

You can configure MongoDB only as a source.

To configure the connection details for MongoDB in your Micro-Integration, see MongoDB Bulk CDC Connection Parameters.

You must also define one Micro-Integration Flow that specifies:

  • the endpoints where data is consumed from and published to. For more information, see Micro-Integration Flow Parameters.

  • how the header fields should be mapped. For more information, see Mapping Headers and Payloads.

    For bulk CDC Micro-Integrations, payload transformations are not supported. Header transformations and Smart Topics are supported.

For message headers, see MongoDB Bulk CDC Message Headers .

MongoDB Bulk CDC Connection Parameters

The following table describes the connection parameters for MongoDB CDC.

Field Description
MongoDB Connection String The connection string URI for MongoDB. Do not include the username and password within the URL. Format: mongodb://host:port/?replicaSet=rsName
Skip Delete Events Select to skip the processing of delete events from the database change stream. The default value is true.
Authentication

The authentication scheme to use when connecting to MongoDB. Choose one of:

Additional Properties

(Optional) Any additional properties for the MongoDB database that apply for all Flows. Click Add Context if you need to specify additional context properties. For more information, see Additional Properties.
Checkpoint Queue The name of the Last Value Queue (LVQ) on the Solace event broker to use for checkpointing. The LVQ stores information about the change events that have been processed and those that are currently being processed. The LVQ must exist on the same event broker and VPN as the target destination.
Auto Create Checkpoint Queue Select to automatically create the LVQ if it does not exist. The default value is false.

Basic Authentication Parameters

Field Description
Username The user to authenticate with MongoDB database.
Password The password to authenticate with MongoDB database.
Auth Source The authentication source database. The default value is admin.

SSL Certificate Authentication Parameters

Field Description
Validate Certificate Hostnames (recommended) Select to validate hostname against server certificate's hostname. The default value is true.
SSL Keystore The keystore file containing the client certificate.
Keystore Type The type of the SSL keystore.
Keystore Password The password for the SSL keystore.
SSL Truststore The truststore file containing trusted certificates.
Truststore Type The type of the SSL truststore.
Truststore Password The password for the SSL truststore.
Username The user to authenticate with MongoDB.
Password The password to authenticate with MongoDB database.
Auth Source The authentication source database. The default value is admin.

Micro-Integration Flow Parameters

You must configure the endpoint parameters for the Flow. The Flow shares the connection details of the parent Micro-Integration.

The following table describes the parameters for configuring MongoDB CDC as a source.

Field Description
Destination

The collections from which to consume change events. Each entry must include the database name and the collection name in <database>.<collection> format. You can specify multiple collections using a semicolon-separated list, a regular expression, or a combination of both. For example:

  • Semicolon-separated list: inventory.customers;inventory.orders

  • Regular expression: inventory.collection_.*

  • Combination: inventory.orders_.*;inventory.customers_.*

Additional Properties

(Optional) Any additional configuration for the MongoDB database that applies for this Flow. Click Add Context if you need to specify additional context properties. For more information, see Additional Properties.

Additional Properties

Config Option Type Valid Values Default Value Description
snapshotMode String initial, schema_only, when_needed, never, no_data, initial_only no_data The criteria for running a snapshot upon startup of the connector. The initial (default) option runs a snapshot that captures the current full state of the configured collections if the connector does not detect any offsets for the logical server name. After the snapshot completes, the connector begins to stream changes from the oplog. The never or no_data options do not run a snapshot. Upon first startup, the Micro-Integration immediately begins reading from the beginning of the oplog, but it does not create READ events to represent the data set at the point of the connector's start-up.
skippedOperations String c, u, d, t, none t A comma-separated list of operation types to skip during streaming. Valid options include: c for inserts/create; u for updates; d for deletes, t for truncates, and none to indicate nothing skipped.
offsetFlushIntervalMs Integer 60000 The interval for flushing offsets in milliseconds.
maxQueueSize Integer 255 The maximum size of the queue for change events read from the database log but not yet recorded or forwarded. This value should always be larger than the maximum batch size.
maxBatchSize Integer 127 The maximum size of each batch of source records.