Migrating to the 4.x Line

You can migrate an existing deployment of the Solace Connector for Apache Spark from the 3.x line to the 4.x line. The configuration of both versions of the connector is the same, so you don't have to update your connector configuration to move to the 4.x line.

  • Solace recommends the 4.x line for new deployments; the 3.x line remains necessary only if your environment is fixed at Apache Spark 3.5.x.

  • Solace supports the 3.x line until approximately September 2027.

3.x and 4.x Line Comparison

Component 3.x Line 4.x Line

Maven coordinate

com.solacecoe.connectors:pubsubplus-connector-spark

com.solacecoe.connectors:pubsubplus-connector-spark

Apache Spark

3.5.x

4.0.x

Scala

2.12

2.13

Java

11

17

Migrating

To migrate your connector:

  • Upgrade your Spark environment to Java 17, Scala 2.13, and Apache Spark 4.0.x.

  • Confirm your Spark environment supplies Spark, Scala, Hadoop, and logging dependencies at runtime, because the 4.x JAR file no longer bundles them.

  • Remove the 3.x JAR file from your classpath before installing the 4.x JAR file, and restart your Spark environment.

A leftover 3.x (Scala 2.12) JAR file on the classpath causes a NoSuchMethodError, ClassNotFoundException, or similar error at startup, because it is binary-incompatible with the 4.x line's Scala 2.13 runtime.

Apache Spark 4.0 enables ANSI SQL mode by default. As a result, Spark raises an error when a query contains an invalid cast, an arithmetic overflow, or a division by zero, where earlier versions returned NULL. This change comes from Apache Spark, not from the connector, but it can cause your existing pipelines to fail after you migrate. To restore the earlier behavior, set spark.sql.ansi.enabled to false in your Spark configuration. In Databricks, use the Spark config field on the Spark tab under Advanced in the cluster configuration.