Micro-Integration for Databricks Zerobus (Beta)

The Solace Micro-Integration for Databricks Zerobus bridges data between a Solace event broker and Databricks Zerobus. The Micro-Integration for Databricks Zerobus provides you (as an operator) a flexible and efficient way to load Databricks Zerobus application data from your Solace-backed, event-driven architecture and the event mesh. The Micro-Integration is deployable standalone or in active-standby redundancy mode to allow for high availability of your data movement. Each instance of this Micro-Integration supports up to 20 individual, concurrent workflows (queue-to-table pipelines), minimizing the number of Micro-Integration instances deployed and managed.

Before You Begin

Before you configure the Micro-Integration for Databricks Zerobus, ensure you meet these requirements:

  • On Azure, use a Databricks Premium workspace. The Standard plan does not support Unity Catalog or Zerobus Ingest.

  • On AWS, ensure that Unity Catalog is enabled for your workspace. New AWS workspaces have Unity Catalog enabled by default.

  • Deploy your workspace and target table in a Zerobus-supported region. For supported regions, see AWS feature region support or Azure feature region support. Both must be in the same supported region.

  • Use a Unity Catalog managed Delta table as your target. Workspace default storage (DBFS root) and external tables are not supported.

  • Do not use a workspace where the target storage is secured through a private endpoint. Private endpoint storage is not supported.

  • Create the target Delta table before starting the Micro-Integration. The Micro-Integration does not create tables automatically.

  • Do not recreate or drop the target Delta table while the Micro-Integration is running. Recreating a target table is not supported.

  • Ensure your network allows outbound TLS 1.2 or higher connections. All communication between the Micro-Integration and Databricks is encrypted.

  • Do not place a proxy or middleware between the Micro-Integration and Databricks. Proxies that strip HTTP/2 headers will break the gRPC connection.

  • Plan for at-least-once message delivery. The Micro-Integration does not deduplicate messages. If your application requires exactly-once semantics, implement deduplication at the application level.

  • Add new columns to the target table before sending records with new fields. The Micro-Integration does not update table schema automatically.

  • Send message payloads as JSON. Databricks Zerobus does not support XML payload format.

  • Use only ASCII letters, digits, and underscores in table and column names.

  • Do not use a compliance security profile workspace. Workspaces with FedRAMP, HIPAA, or PCI-DSS compliance security profiles are not supported.

  • We strongly recommend configuring a Dead Message Queue (DMQ) on the event brokerevent broker service queue before deploying the Micro-Integration. When a message cannot be delivered to Databricks, the Micro-Integration NACKs it and the event broker sends it to the DMQ so it can be inspected and reprocessed. Without a DMQ, NACKed messages are permanently lost.

Configuring the Micro-Integration

Spring Framework technologies make it easy to configure the Micro-Integration, perform advanced logging capabilities, and export live metric data to external monitoring solutions.

To get started deploying a Micro-Integration locally or within a container, see:

For common configuration details, see:

For configuration settings that are specific to Databricks Zerobus, see: