Serialization and Deserialization with Solace Schema Registry

Serialization means converting an application's data objects from their native format into a binary format suitable for transmission or storage. For example, a serializer might take a Java object and convert it to Avro binary format using a schema from the registry.

Deserialization is converting the binary data back into its original format for application processing.

Serialization and deserialization of messages with Solace Messaging APIs is supported by the SERDES Collection. These APIs allow you to specify the schema you want to use to perform SERDES operations on Solace event messages. Each different API and payload format pair needs its own SERDES.

You deploy the SERDES Collection alongside the Solace Messaging API for the programming language and schema format you are using to develop your application.

The following topics provide language-agnostic information about SERDES configuration and concepts that apply across all messaging protocols and programming languages.

Using the SERDES Collection with Solace Schema Registry

In modern messaging systems, managing data formats across distributed applications can present challenges. As you scale your applications, differences in how message producers and consumers serialize and interpret data can lead to issues with data consistency and version compatibility. Without a shared understanding of the message format, mismatches such as missing fields, unexpected data types, or incompatible versions can result in runtime errors. To prevent these problems and streamline data exchange in Solace messaging environments, we recommend you use Solace Schema Registry with the SERDES Collection.

Understanding schemas

A schema defines the structure, data types, and validation rules for messages exchanged between producers and consumers, ensuring that they agree on the format of the data being exchanged. In formats like Avro or JSON Schema, a schema explicitly describes each field's name, type, and if it is required. This enables compatibility checks, validation, and tooling support, and ensures consistent data interpretation across services, especially when schemas are managed in a schema registry.

Understanding the details of Schema Registry

A schema registry is a centralized service that stores and manages schemas used for serializing and deserializing structured data. It ensures that applications exchanging messages follow a consistent data format, which prevents compatibility issues between message producers and consumers. By using Solace Schema Registry, you can enforce data validation, track schema versions, and support schema evolution in a way that maintains compatibility with applications using older schema versions.

Understanding SERDES and Schema Registry Interactions and Workflows

When integrated with the SERDES Collection, your applications can automatically retrieve and apply the correct schema from Solace Schema Registry during message serialization and deserialization. When a message producer sends a message, the serializer component encodes the data according to a registered schema, including a schema identifier stored in the SERDES header appended to your message. On the message consumer side, the application extracts the schema identifier from the header, retrieves the corresponding schema from the registry, and ensures the message is correctly parsed. This header-based approach allows for efficient schema resolution without modifying the message payload itself. This interaction enables dynamic schema resolution, eliminates the need to hardcode schemas in applications, and simplifies the management of structured data formats like Avro and JSON Schema in your messaging applications.

Steps for Using the SERDES Collection with Solace Schema Registry

  1. Set up a Solace Schema Registry instance then define and register your schemas.
  2. Choose a serialization format. Solace currently supports Avro and JSON Schema.
  3. Configure your SERDES objects (Avro or JSON Schema).
    • Both JSON and Avro share a common SERDES configuration API for registry access, resolution, and caching.
  4. Use a serializer to serialize outbound messages and a deserializer to deserialize inbound messages.

For more information, see Solace Schema Registry.

API and Protocol-Specific Implementation Guides

The following pages provide protocol-specific implementation details, including code examples and language-specific considerations for using SERDES with different messaging APIs and protocols:

Java:

.NET:

Getting Started with SERDES

Before configuring advanced features like resolution strategies or performance optimization, you need to connect to your Schema Registry instance and understand basic SERDES configuration. This section covers the essential setup steps to get started.

All SERDES properties are set in a configuration map. For protocol-specific examples of how to create and configure this map, see API and Protocol-Specific Implementation Guides.

Connecting to Schema Registry

The Generic SERDES for Java supports multiple authentication methods for connecting to Solace Schema Registry. This section covers how to configure authentication credentials for your SERDES objects, including basic username and password authentication, and how to implement them in both secure and non-secure connection contexts. You specify the authentication settings when you configure your schema registry client, which allows you to control access permissions and maintain the security of your schemas.

The following properties configure how your application connects to Solace Schema Registry:

  • solace.registry.url—The URL of Solace Schema Registry where schemas are stored. This property is required in all serializer and deserializer configurations. The registry URL must point to the Registry REST API endpoint. The default path for this endpoint is /apis/registry/v3.
  • solace.registry.auth.username—The username to use to authenticate with Solace Schema Registry.
  • solace.registry.auth.password—The password to use to authenticate with Solace Schema Registry.
  • solace.registry.trust-store.path—(Java only) The file path to the truststore containing SSL/TLS certificates for secure connections.
  • solace.registry.trust-store.password—(Java only) The password required to access the truststore file.
  • solace.registry.trust-store.certificates—(.NET only) Specifies the collection of SSL/TLS certificates for secure connections.
  • solace.registry.validate-certificate—A flag that determines whether SSL certificate validation is enabled (true) or disabled (false).

    We recommend that you never disable solace.registry.validate-certificate in production environments because it creates a security vulnerability.

  • solace.registry.validate-certificate-hostname—(.NET only) A flag that determines whether SSL certificate hostname validation is enabled (true) or disabled (false) when connecting to Schema Registry over HTTPS.

The following sections describe how to set these properties based on which authentication option you use:

  • Authentication, plain text connection—Used in restricted environments where network access is controlled but authentication is still required. Set solace.registry.url to an HTTP endpoint (for example, http://localhost:8081/apis/registry/v3), and provide solace.registry.auth.username and solace.registry.auth.password credentials.
  • Authentication, secure connection—The recommended production setup. This setup ensures both data encryption and access control for secure schema management. Set solace.registry.url to an HTTPS endpoint (for example, https://{your_hostname}:443/apis/registry/v3), provide solace.registry.auth.username and solace.registry.auth.password credentials, configure solace.registry.trust-store.path and solace.registry.trust-store.password for TLS, and set solace.registry.validate-certificate to true.
  • Advanced authentication—Used when you configure Schema Registry with an external identity provider such as Microsoft Entra ID. When using advanced authentication flows, you configure the same solace.registry.auth.username and solace.registry.auth.password properties. In this case, solace.registry.auth.username maps to your OAuth Client ID, and solace.registry.auth.password maps to your OAuth Client Secret from your OAuth configuration. For more information about configuring external identity providers with Schema Registry, see Deploying and Configuring Solace Schema Registry with Docker or Podman.

For protocol-specific code examples showing how to configure these properties, see API and Protocol-Specific Implementation Guides.

Basic Configuration Setup

All SERDES configuration properties are set in a configuration map object. The specific implementation varies by protocol and programming language, but the concept remains consistent across all implementations. You create a configuration map, populate it with the necessary properties (such as registry URL, authentication credentials, and format-specific settings), and then use this map to configure your serializer and deserializer objects.

  • solace.registry.schema-resolver—Allows configuration of a custom schema resolver implementation for advanced schema resolution scenarios beyond the standard artifact resolver strategies. This property provides fine-grained control over how schemas are retrieved and cached.

For detailed examples of how to create and configure the configuration map for your specific protocol, see API and Protocol-Specific Implementation Guides. These pages provide complete code examples showing how to instantiate the configuration map, set properties, and configure serializers and deserializers.

Choosing Schema Resolution Strategies

Artifact resolver strategies determine how the serializer selects which schema to use when serializing messages. The strategy you choose depends on your application architecture, topic structure, and how you organize schemas in your registry. This section helps you understand the available strategies and choose the right one for your use case.

Understanding Resolution Strategies

Artifact resolver strategies define how a schema or artifact is dynamically resolved at runtime for serialization operations performed by the SERDES Collection. These strategies determine which schema to fetch from Solace Schema Registry based on metadata like Solace topics, destination IDs, and record IDs. Artifact resolver strategies are only applied during serialization, because the serializer is responsible for selecting or creating the artifact ID to register or reference a schema. Deserializers simply use the schema ID embedded in the incoming message to retrieve the correct schema, without needing to determine how that ID was formed.

Key points about resolution strategies:

  • Strategies only apply during serialization (deserializers use the schema ID from the message)
  • The strategy determines how to construct an ArtifactReference from message metadata
  • Different strategies offer different levels of flexibility and complexity

Some important terms:

  • record—A structured data object, for example an Avro or JSON Schema message, that is serialized or deserialized using a schema from Solace Schema Registry.
  • artifact—A named and versioned container in Solace Schema Registry that holds one or more schema versions. Each artifact is identified by an artifactId and optionally grouped using a groupId.
  • ArtifactReference—A pointer to an existing artifact in Solace Schema Registry. When the artifact contains a schema, this reference can be used to locate and apply it during serialization or deserialization.
  • artifactId—A unique identifier for an artifact in Solace Schema Registry, which typically contains a single schema.
  • groupId—A logical grouping mechanism for artifacts in Solace Schema Registry. It allows organizing related schemas, for example all schemas for a specific application. If no groupId is specified, the value is default.
In cases where multiple topics should resolve to the same schema, Solace recommends you use the Solace Topic ID Strategy with Profiles. This approach provides flexible topic-to-schema mappings using wildcard expressions and custom artifact references.

Strategy Comparison

The following table compares the available resolution strategies to help you choose the right one for your use case:

Strategy Best For
Destination ID Simple destination-to-schema mapping
Solace Topic ID Direct topic name to schema mapping
Topic ID with Profiles Multiple topics sharing schemas, wildcard patterns
Record ID (Avro) Schema information embedded in record

Destination ID Strategy

Use the Destination ID Strategy when you have a simple one-to-one mapping between message destinations and schemas. This works well when each REST endpoint, AMQP queue, or messaging destination consistently uses the same schema, and your schema registry follows a naming convention where artifact IDs match your destination names.

The DestinationIdStrategy automatically determines which schema to use during serialization. It extracts the destination name from the record metadata to use as the artifactId, while applying a default value as the groupId to construct the complete ArtifactReference. This approach eliminates the need for explicit schema specification on a per message basis because it creates a direct mapping between message destinations and schema artifacts stored in the registry. For successful implementation, Solace Schema Registry must contain schemas with artifactIds that correspond to the destination names used in your application's messaging infrastructure. The strategy is valuable in systems where different message types are routed to different destinations, as it reduces configuration overhead and decouples serialization logic from schema-specific details.

In summary, the DestinationIdStrategy:

  • Uses the destination name defined by the parameter passed into the serialize() method call. In REST and AMQP implementations, this is typically an endpoint name or path specified directly by the user. In Solace JCSMP API implementations, this destination is automatically extracted from the message topic via the serialize() method.
  • Uses this destination name as the artifactId in the ArtifactReference.
  • Uses a default value as the groupId in the ArtifactReference.
  • Uses the constructed ArtifactReference to locate the correct schema in Solace Schema Registry.
This strategy requires that schemas are registered in Solace Schema Registry with artifactIds that match the destination names used in your application.

To configure a serializer to use the DestinationIdStrategy, set the solace.registry.artifact-resolver-strategy property to DestinationIdStrategy.class. For protocol-specific code examples, see API and Protocol-Specific Implementation Guides.

Solace Topic ID Strategy

Use the Solace Topic ID Strategy when your Solace topic names directly correspond to schema artifact IDs in your registry. This is ideal for Solace applications where topic hierarchies are well-defined and schema naming follows the same structure.

The SolaceTopicIdStrategy automatically determines which schema to use during serialization by mapping the destination name directly to the ArtifactReference, setting the destination string as the artifactId. This approach simplifies schema resolution by creating a direct correlation between Solace topic destinations and schema artifacts stored in the registry. For successful implementation, Solace Schema Registry must contain schemas with artifactIds that correspond to the topic names used in your Solace messaging infrastructure.

In summary, the SolaceTopicIdStrategy:

  • Uses the destination name defined by the parameter passed into the serialize() method call. In REST and AMQP implementations, this is typically an endpoint name or path specified directly by the user. In Solace JCSMP API implementations, this destination is automatically extracted from the message topic via the serialize() method.
  • Maps this destination name directly as the artifactId in the ArtifactReference.
  • Uses the constructed ArtifactReference to locate the correct schema in Solace Schema Registry.
This strategy requires that schemas are registered in Solace Schema Registry with artifactIds that match the topic names used in your Solace messaging application.

To configure a serializer to use the SolaceTopicIdStrategy without a profile, set the solace.registry.artifact-resolver-strategy property to SolaceTopicIdStrategy.class. For protocol-specific code examples, see API and Protocol-Specific Implementation Guides.

Solace Topic ID Strategy with Profiles

Use Topic ID Strategy with Profiles when you need flexible topic-to-schema mappings, such as when multiple topics should use the same schema, or when you want to use wildcard patterns to map topic hierarchies to schemas. This strategy is the most flexible and is recommended when you have complex topic structures or need to map many topics to fewer schemas.

For more advanced mapping scenarios, the SolaceTopicIdStrategy can be used with a SolaceTopicProfile to provide flexible topic-to-schema mappings. This approach allows for wildcard topic expressions and custom artifact references, giving you greater control over schema resolution.

Using profiles with SolaceTopicIdStrategy provides several benefits:

  • Support for wildcard topic expressions to match multiple topics with a single mapping. You can use * for single-level wildcards and > for multi-level wildcards. For more information, see Wildcard Characters in Topic Subscriptions.
  • The ability to map different topic patterns to specific schemas.
  • Control over groupId, artifactId, and versioning for schema selection

The SolaceTopicProfile can be configured with different types of mappings:

Topic Expression Only Mapping

This method maps a Solace topic expression directly to an artifactId that matches the topic expression itself. The groupId is left as default. Use this method when your schema registry follows a convention where schemas are registered with an artifactId that matches your topic hierarchy.

To implement this mapping:

  1. Create a new instance of SolaceTopicProfile
  2. Create mappings with topic expressions using SolaceTopicArtifactMapping.create() with patterns like:
    • Exact match: "solace/samples"
    • Single-level wildcard: "solace/*/sample"
    • Multi-level wildcard: "solace/>"
  3. Add each mapping to the profile
  4. Set the solace.registry.strategy-topic-profile property to the configured profile

For protocol-specific code examples, see API and Protocol-Specific Implementation Guides.

Topic Expression with Custom Artifact ID

This method maps a Solace topic expression to a specific, custom artifactId. The groupId is left as "default". Use this method when multiple topics should use the same schema, but your topic names do not match your schema registry naming conventions.

To implement this mapping:

  1. Create a new instance of SolaceTopicProfile
  2. Create mappings with topic expressions and custom artifact IDs using SolaceTopicArtifactMapping.create(topicExpression, artifactId). For example:
    • Map "solace/samples" to artifact ID "User"
    • Map "solace/*/sample" to artifact ID "NewUser"
    • Map "solace/>" to artifact ID "OldUser"
  3. Add each mapping to the profile
  4. Set the solace.registry.strategy-topic-profile property to the configured profile

For protocol-specific code examples, see API and Protocol-Specific Implementation Guides.

Topic Expression with Full ArtifactReference

This method maps a Solace topic expression to a complete ArtifactReference with a custom groupId, artifactId, and optional version for exact schema selection. Use this method when you need complete control over schema resolution, especially when you have multiple schema groups or when specific versions must be used.

To implement this mapping:

  1. Create a new instance of SolaceTopicProfile
  2. Use an ArtifactReferenceBuilder to create artifact references with:
    • groupId (for example, "com.solace.samples.serdes.avro.schema")
    • artifactId (for example, "User")
    • Optional version (for example, "0.0.1") for version-specific references
  3. Create mappings using SolaceTopicArtifactMapping.create(topicExpression, artifactReference)
  4. Add each mapping to the profile
  5. Set the solace.registry.strategy-topic-profile property to the configured profile

For protocol-specific code examples, see API and Protocol-Specific Implementation Guides.

Record ID Strategy (Avro Only)

The RecordIdStrategy automatically determines which schema to use during serialization by extracting information directly from the record's payload. It uses the schema name as the artifactId and the schema namespace as the groupId to construct the complete ArtifactReference. This approach eliminates the need for explicit schema specification on a per message basis because it creates a direct mapping between the data structure of your records and schema artifacts stored in the registry. For successful implementation, Solace Schema Registry must contain schemas with an artifactId and a groupId that corresponds to the schema name and namespace used in your application's data models. The strategy is valuable in systems where the schema information is inherently contained within the record structure, as it reduces configuration overhead and ensures that the correct schema is always used for each record type.

Use the Record ID Strategy when your Avro records contain schema information (name and namespace) that should determine which schema to use. This strategy is valuable when the schema information is inherently part of your data model and you want the serializer to automatically extract it from each record.

In summary, the RecordIdStrategy:

  • Extracts the schema from the record's payload.
  • Uses the schema name as the artifactId in the ArtifactReference.
  • Uses the schema namespace as the groupId in the ArtifactReference.
  • Uses the constructed ArtifactReference to locate the correct schema in Solace Schema Registry.
This strategy requires that schemas be registered in Solace Schema Registry with artifactIds that match the schema names and groupIds that match the schema namespaces used in your application.

To configure a serializer to use the RecordIdStrategy, set the solace.registry.artifact-resolver-strategy property to RecordIdStrategy.class. For protocol-specific code examples, see API and Protocol-Specific Implementation Guides.

Optimizing Performance

Once your SERDES implementation is working, you can optimize performance through caching and lookup configuration. These settings control how frequently your application queries the schema registry and how it handles schema resolution, allowing you to balance performance, freshness, and resilience.

Schema Caching Options

To reduce the overhead of repeated schema registry lookups, clients using the SERDES Collection can configure schema caching options. These settings control how long schemas are stored locally and how different types of lookups interact with the schema registry cache.

  • solace.registry.cache.artifact-ttl-ms—Determines how long schema artifacts remain valid in the cache before they need to be fetched again from the registry on the next relevant lookup. The default value is 30000ms (30 seconds). Longer TTLs improve performance by reducing registry calls but risk using outdated schemas, shorter TTLs ensure fresher schemas but increase registry load. A zero TTL disables caching entirely, requiring registry fetches for every request. This property accepts:
    • Long value (for example, 5000L for 5 seconds)
    • String value (for example, "5000")
    • Duration object (for example, 5 seconds)
    • Zero (0L) to disable caching completely
  • solace.registry.cache.use-cached-on-error—Controls whether to use cached schemas when schema registry lookup errors occur. When enabled (true), schema resolution uses cached schemas instead of throwing exceptions after retry attempts are exhausted, improving resilience during registry outages. When disabled (false, the default), the API throws an exception when registry lookup errors occur.
  • solace.registry.cache.cache-latest (Serializers Only)—Controls whether schema lookups that specify latest or do not include an explicit version (no-version) create additional cache entries, so that future "latest" or versionless lookups can be resolved using the cache instead of querying the registry again. When enabled (true, the default), latest and versionless (no-version) schema lookups create additional cache entries, enabling subsequent latest or no-version lookups to use the cached schema without contacting the registry. When disabled (false), only the resolved version is cached, meaning every latest or versionless lookup must query the registry to determine the current latest version.
    • When an explicit schema version is specified, the solace.registry.cache.cache-latest setting has no effect on schema lookup results.
    • solace.registry.cache.cache-latest only affects caching for serialization operations. It does not apply to schema references, which are always resolved through direct registry lookups, bypassing the cache.

For protocol-specific code examples showing how to configure these properties, see API and Protocol-Specific Implementation Guides.

Schema Lookup Options

When you use the SERDES Collection with Solace Schema Registry, the serializer must determine which schema to use when writing data. You can configure schema lookup options to control how the appropriate schema is resolved from the registry. The following schema lookup options are available:

  • solace.registry.find-latest-artifact—A boolean flag that determines whether your serializer should attempt to locate the most recent artifact in the registry for the given groupId and artifactId combination. The default value is false.
  • solace.registry.explicit-artifact.version—A string value that represents the artifact version used for querying an artifact in the registry (for example, "1.0.0"). This property overrides the version returned by the ArtifactReferenceResolverStrategy. If this property and solace.registry.find-latest-artifact are both set, this property takes precedence. The default value is null.
  • solace.registry.explicit-artifact.artifact-id—A string value that represents the artifactId used for querying an artifact in the registry (for example, "my-schema"). This property overrides the artifactId returned by the ArtifactReferenceResolverStrategy. The default value is null.
  • solace.registry.explicit-artifact.group-id—A string value that represents the groupId used for querying an artifact in the registry (for example, "com.example"). This property overrides the groupId returned by the ArtifactReferenceResolverStrategy. The default value is null.
  • solace.registry.request.attempts—Specifies the number of attempts to make when communicating with the schema registry before giving up. Valid values are any number between 1 and Long.MAX_VALUE. When used with solace.registry.cache.use-cached-on-error, this property specifies the number of attempts before falling back to the last cached value. The default value is 3.
  • solace.registry.request.attempt-backoff-ms—Specifies the backoff time in milliseconds between retry attempts when communicating with the schema registry. Valid values are any number between 0 and Long.MAX_VALUE. This property accepts:
    • Long value (for example, 500L)
    • String value (for example, "500")
    • Duration object (for example, 500 milliseconds)
    The default value is 500 milliseconds.

For protocol-specific code examples showing how to configure these properties, see API and Protocol-Specific Implementation Guides.

Additional SERDES Resources

For additional SERDES configuration options and reference information, see the following pages: