Databricks Zerobus (Beta)

You can configure Databricks Zerobus only as a target.

To configure the connection details for Databricks Zerobus in your Micro-Integration, see Databricks Zerobus Connection Parameters.

You must also define at least one Micro-Integration Flow that specifies:

the endpoints where data is consumed from and published to. For more information, see Micro-Integration Flow Parameters.
whether Flow-level error handling is configured. For more information, see Error Handling for Micro-Integration Flows.
how the header and payload fields should be mapped. For more information, see Mapping Headers and Payloads.

For message headers, see Databricks Zerobus Message Headers.

Before You Begin

Before you configure the Databricks Zerobus Micro-Integration, ensure you meet these requirements:

On Azure, use a Databricks Premium workspace. The Standard plan does not support Unity Catalog or Zerobus Ingest.
On AWS, ensure that Unity Catalog is enabled for your workspace. New AWS workspaces have Unity Catalog enabled by default.
Deploy your workspace and target table in a Zerobus-supported region. For supported regions, see AWS feature region support or Azure feature region support. Both must be in the same supported region.
Use a Unity Catalog managed Delta table as your target. Workspace default storage (DBFS root) and external tables are not supported.
Do not use a workspace where the target storage is secured through a private endpoint. Private endpoint storage is not supported.
Create the target Delta table before starting the Micro-Integration. The Micro-Integration does not create tables automatically.
Do not recreate or drop the target Delta table while the Micro-Integration is running. Recreating a target table is not supported.
Ensure your network allows outbound TLS 1.2 or higher connections. All communication between the Micro-Integration and Databricks is encrypted.
Do not place a proxy or middleware between the Micro-Integration and Databricks. Proxies that strip HTTP/2 headers will break the gRPC connection.
Plan for at-least-once message delivery. The Micro-Integration does not deduplicate messages. If your application requires exactly-once semantics, implement deduplication at the application level.
Add new columns to the target table before sending records with new fields. The Micro-Integration does not update table schema automatically.
Send message payloads as JSON. Databricks Zerobus does not support XML payload format.
Use only ASCII letters, digits, and underscores in table and column names.
Do not use a compliance security profile workspace. Workspaces with FedRAMP, HIPAA, or PCI-DSS compliance security profiles are not supported.
We strongly recommend configuring a Dead Message Queue (DMQ) on the event broker service queue before deploying the Micro-Integration. When a message cannot be delivered to Databricks, the Micro-Integration NACKs it and the event broker sends it to the DMQ so it can be inspected and reprocessed. Without a DMQ, NACKed messages are permanently lost.

Setting Up a Service Principal

The Databricks ZerobusMicro-Integration uses OAuth 2.0 service principal credentials to authenticate.

To obtain OAuth 2.0 credentials for the Micro-Integration, create a Databricks-managed service principal. Do not use an Entra ID or Azure AD service principal.

In the Databricks workspace, go to Settings > Identity and access > Service principals > Manage.
Click + Add service principal > Add new, enter a name for the service principal, then click Add.
On the Secrets (or Credentials) tab, click Generate secret.

The Client Secret is shown only once. Copy and save it immediately to a secure location. It cannot be retrieved after you close the dialog.
On the Configurations tab, copy the Application ID (UUID). This is your Client ID.
Grant the service principal the minimum required privileges on the target Unity Catalog table. Run the following SQL in a Databricks notebook or SQL editor, replacing the placeholders with your catalog, schema, table, and service principal UUID:
```
GRANT USE CATALOG ON CATALOG <catalog>                        TO `<service-principal-UUID>`;
GRANT USE SCHEMA  ON SCHEMA  <catalog>.<schema>               TO `<service-principal-UUID>`;
GRANT MODIFY, SELECT ON TABLE <catalog>.<schema>.<table>      TO `<service-principal-UUID>`;
```
MODIFY and SELECT must both be granted explicitly, even if ALL PRIVILEGES was previously granted at a higher level.

For more information, see:

Manage service principals and Authorize service principal access with OAuth in the AWS Databricks documentation
Manage service principals and Authorize service principal access to Azure Databricks with OAuth in the Azure Databricks documentation

Finding Your Connection Values

To find your Workspace URL and Server Endpoint, open your Databricks workspace in a browser. The full URL has this format:

https://<workspace-domain>/?o=<workspace-id>

Workspace URL: The base URL before /?o=. For example:
- AWS: https://<id>.cloud.databricks.com
- Azure: https://adb-<workspace-id>.<suffix>.azuredatabricks.net
Workspace ID: The numeric value after o= in the full browser URL.
Server Endpoint: Constructed from the workspace ID and region. For example:
- AWS: https://<workspace-id>.zerobus.<region>.cloud.databricks.com
- Azure: https://<workspace-id>.zerobus.<azure-region>.azuredatabricks.net

Databricks Zerobus Connection Parameters

The following table describes the connection parameters for Databricks Zerobus.

Field	Description
Databricks Workspace URL	The URL of the Databricks workspace. For example, `https://<id>.cloud.databricks.com` for AWS, or `https://<instance>.azuredatabricks.net` for Azure. For more information, see Databricks Zerobus Connection Parameters.
Zerobus Server Endpoint	The Databricks Zerobus gRPC server endpoint. For more information, see Finding Your Connection Values.
Client ID	The OAuth 2.0 service principal client ID (Application ID). For more information, see Setting Up a Service Principal.
Client Secret	The OAuth 2.0 service principal client secret. Treat this value as a secret. For more information, see Setting Up a Service Principal.

Micro-Integration Flow Parameters

You must configure the endpoint parameters for each Flow. Each Flow can have different settings, but they all share the connection details of the parent Micro-Integration.

Databricks Zerobus Target Parameters

The following table describes the parameters for configuring Databricks Zerobus as a target.

Setting a Smart Topic Destination on the Mappings step overrides the target destination field set on the Details step.

Field	Description
Catalog	The Unity Catalog name (first segment of the fully-qualified table name: `catalog.schema.table`).
Schema	The schema within the catalog (second segment of the fully-qualified table name: `catalog.schema.table`).
Table	The target Delta table name (third segment of the fully-qualified table name: `catalog.schema.table`).

Monitoring with Databricks System Tables

Databricks provides monitoring tools to help you track the health and usage of your ingestion pipelines. For more information, see:

AWS: Monitor your Databricks Zerobus usage
Azure: Monitor your Databricks Zerobus usage

Troubleshooting

The following troubleshooting tips might help you to resolve issues with this Micro-Integration. If problems persist, contact Solace.

We strongly recommend configuring a Dead Message Queue (DMQ) on the event broker service queue before deploying the Micro-Integration. When a message cannot be delivered to Databricks, the Micro-Integration NACKs it and the event broker sends it to the DMQ so it can be inspected and reprocessed. Without a DMQ, NACKed messages are permanently lost.

Authentication Errors

If the Micro-Integration logs an authentication error or fails to connect:

Verify that the Client ID and Client Secret are correct. You can check them on the service principal's Configurations and Secrets tabs in the Databricks workspace.
Check that the service principal's client secret has not expired. Databricks client secrets have a configurable lifetime. If the secret has expired, generate a new one and update the Micro-Integration credentials.
Confirm that the service principal has not been deleted or disabled in the Databricks workspace.
Ensure that the service principal has the required privileges on the target Unity Catalog table. For more information, see Setting Up a Service Principal.

Table Not Found Errors

If the Micro-Integration logs errors indicating that the target table cannot be found:

Verify that the Catalog, Schema, and Table values in the flow configuration match the fully-qualified name of an existing Delta table.
Confirm that the target table exists in the Databricks workspace. The Micro-Integration does not create tables automatically.
Check that the service principal has USE CATALOG, USE SCHEMA, MODIFY, and SELECT privileges on the target table. For more information, see Setting Up a Service Principal.

Schema Mismatch or Unknown Column Errors

If the Micro-Integration logs errors about unrecognized or unexpected columns:

Check that the message payload does not contain fields absent from the Delta table schema. Add missing columns to the table before sending records that include those fields. The Micro-Integration does not update table schema automatically.
Ensure that all column names use only ASCII letters, digits, and underscores.

gRPC Connection Failures

If the Micro-Integration logs gRPC connection errors or transport-level failures:

Check that no proxy or middleware is placed between the Micro-Integration and Databricks. Proxies that strip HTTP/2 headers will break the gRPC connection.
Verify that your network allows outbound TLS 1.2 or higher connections to the Zerobus server endpoint.
Confirm that both the Databricks workspace and the target table are deployed in a region that supports Zerobus Ingest. For supported regions, see AWS feature region support or Azure feature region support. Both must be in the same supported region.

Micro-Integration Stops Working After Running Correctly

If a previously working Micro-Integration suddenly stops:

Check that the service principal's client secret has not expired. Databricks client secrets have a configurable lifetime. If expired, generate a new secret and update the Micro-Integration credentials.
Verify that the service principal still exists and is active in the Databricks workspace.
Ensure that the service principal still has the required privileges on the target table. A workspace admin can revoke privilege grants.

Provide feedback