SynapseML

Synapse Machine Learning

SynapseML (previously known as MMLSpark) is an open-source library that simplifies the creation of massively scalable machine learning (ML) pipelines. SynapseML provides simple, composable, and distributed APIs for a wide variety of different machine learning tasks such as text analytics, vision, anomaly detection, and many others. SynapseML is built on the Apache Spark distributed computing framework and shares the same API as the SparkML/MLLib library, allowing you to seamlessly embed SynapseML models into existing Apache Spark workflows.

With SynapseML, you can build scalable and intelligent systems to solve challenges in domains such as anomaly detection, computer vision, deep learning, text analytics, and others. SynapseML can train and evaluate models on single-node, multi-node, and elastically resizable clusters of computers. This lets you scale your work without wasting resources. SynapseML is usable across Python, R, Scala, Java, and .NET. Furthermore, its API abstracts over a wide variety of databases, file systems, and cloud data stores to simplify experiments no matter where data is located.

SynapseML publishes runtime-specific JVM artifacts: Spark 3.5 uses Scala 2.12, while Spark 4.0 and 4.1 use Scala 2.13. See the installation matrix before selecting a Maven coordinate.

Topics	Links
Build
Version
Docs
Support
Binder
Usage

• Synapse Machine Learning
  • Features
  • Documentation and Examples
  • Setup and installation
    • Microsoft Fabric
    • Synapse Analytics
    • Databricks
    • Python Standalone
    • Spark Submit
    • SBT
    • Apache Livy and HDInsight
    • Docker
    • R
    • Building from source
  • Papers
  • Learn More
  • Contributing & feedback
  • Other relevant projects

Features

Vowpal Wabbit on Spark	The Cognitive Services for Big Data	LightGBM on Spark	Spark Serving
Fast, Sparse, and Effective Text Analytics	Leverage the Microsoft Cognitive Services at Unprecedented Scales in your existing SparkML pipelines	Train Gradient Boosted Machines with LightGBM	Serve any Spark Computation as a Web Service with Sub-Millisecond Latency

HTTP on Spark	ONNX on Spark	Responsible AI	Spark Binding Autogeneration
An Integration Between Spark and the HTTP Protocol, enabling Distributed Microservice Orchestration	Distributed and Hardware Accelerated Model Inference on Spark	Understand Opaque-box Models and Measure Dataset Biases	Automatically Generate Spark bindings for PySpark and SparklyR

Isolation Forest on Spark	CyberML	Conditional KNN
Distributed Nonlinear Outlier Detection	Machine Learning Tools for Cyber Security	Scalable KNN Models with Conditional Queries

Documentation and Examples

For quickstarts, documentation, demos, and examples please see our website.

Setup and installation

SynapseML installation has two parts: the language wrapper and the JVM artifacts loaded by Spark. Installing synapseml from PyPI does not add the JVM artifacts. A Python wrapper can import successfully while its JVM class is missing; using a _2.12 artifact with Spark 4 can produce errors such as LightGBMClassifier does not exist in the JVM.

Choose one complete published build from the Spark runtime. master is the canonical Spark 3.5 development line; Spark 4.0 and Spark 4.1 are maintained on their corresponding branches.

Code line	Spark runtime	Scala	Python baseline	Release tag	Python package	Maven coordinate
master	Spark 3.5.x	2.12	Python 3.11	v1.1.3	synapseml==1.1.3	com.microsoft.azure:synapseml_2.12:1.1.3
spark4.0	Spark 4.0.1+ (<4.1)	2.13	Python 3.12	v1.1.3-spark4.0	synapseml==1.1.3	com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0
spark4.1	Spark 4.1.x	2.13	Python 3.13	v1.1.3-spark4.1	synapseml==1.1.3	com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1

Always configure https://mmlspark.blob.core.windows.net/maven, where the Spark 4 artifacts are published. See the full installation guide for platform-specific details.

Latest master snapshot

The latest successful master build targets Spark 3.5 and Scala 2.12. This copy-ready command reads the current snapshot version published by CI and starts Spark with that exact JVM build:

MASTER_VERSION="$(
  curl -fsSL https://mmlspark.blob.core.windows.net/icons/badges/master_version3.svg |
    sed -n 's/.*aria-label="master version: \([^"]*\)".*/\1/p'
)"
test -n "$MASTER_VERSION"
spark-shell \
  --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.12:${MASTER_VERSION}"

The PyPI package contains released Python wrappers. If you need Python APIs that are new on master, build the matching wheel from source.

First select the correct platform that you are installing SynapseML into:

• Synapse Machine Learning
  • Features
  • Documentation and Examples
  • Setup and installation
    • Latest master snapshot
    • Microsoft Fabric
    • Synapse Analytics
    • Databricks
    • Python Standalone
    • Spark Submit
    • SBT
    • Apache Livy and HDInsight
    • Docker
    • R
    • Building from source
  • Papers
  • Learn More
  • Contributing & feedback
  • Other relevant projects

Microsoft Fabric

In Microsoft Fabric notebooks SynapseML is already installed. The following copy-ready override targets a Spark 4.1 / Scala 2.13 runtime:

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1",
      "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.13,org.scalactic:scalactic_2.13,org.scalatest:scalatest_2.13,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false"
  }
}

Synapse Analytics

Current Azure Synapse pools use Spark 3.5. Place the following in the first cell of your notebook:

%%configure -f
{
  "name": "synapseml",
  "conf": {
      "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.1.3",
      "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
      "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind",
      "spark.yarn.user.classpath.first": "true",
      "spark.sql.parquet.enableVectorizedReader": "false"
  }
}

To install at the pool level instead of the notebook level add the spark properties listed above to the pool configuration.

Databricks

To install SynapseML on the Databricks cloud, create a new library from Maven coordinates in your workspace.

Use one of these exact Maven coordinates:

• Spark 4.1 / Scala 2.13: com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1
• Spark 4.0 / Scala 2.13: com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0
• Spark 3.5 / Scala 2.12: com.microsoft.azure:synapseml_2.12:1.1.3

Add the resolver https://mmlspark.blob.core.windows.net/maven, attach the library to the target cluster, and restart it before importing synapse.ml.

You can use SynapseML in both your Scala and PySpark notebooks. To get started with our example notebooks import the following databricks archive:

https://mmlspark.blob.core.windows.net/dbcs/SynapseMLExamplesv1.1.3.dbc

Python Standalone

Choose exactly one complete runtime variant below, then start Spark with that variant's JVM artifact.

Spark 4.1 / Python 3.13

python -m pip install "synapseml==1.1.3" "pyspark>=4.1,<4.2"

Spark 4.0 / Python 3.12

python -m pip install "synapseml==1.1.3" "pyspark>=4.0.1,<4.1"

Spark 3.5 / Python 3.11

python -m pip install "synapseml==1.1.3" "pyspark>=3.5,<3.6"

from pyspark.sql import SparkSession

# Spark 4.1. Select the coordinate matching the PySpark command used above.
synapseml_coordinate = "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1"
# Spark 4.0:
# synapseml_coordinate = "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0"
# Spark 3.5:
# synapseml_coordinate = "com.microsoft.azure:synapseml_2.12:1.1.3"

spark = (
    SparkSession.builder.appName("MyApp")
    .config("spark.jars.packages", synapseml_coordinate)
    .config(
        "spark.jars.repositories",
        "https://mmlspark.blob.core.windows.net/maven",
    )
    .getOrCreate()
)
import synapse.ml

Spark Submit

SynapseML can be conveniently installed on existing Spark clusters via the --packages option. Each example below is independently copyable.

# Spark 4.1
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.1"

# Spark 4.0
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.13:1.1.3-spark4.0"

# Spark 3.5
pyspark --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.12:1.1.3"

AWS EMR and pre-provisioned clusters

--packages is supported for production Spark jobs as well as notebooks. For example, use it with spark-submit on EMR rather than downloading a single JAR manually:

spark-submit \
  --repositories "https://mmlspark.blob.core.windows.net/maven" \
  --packages "com.microsoft.azure:synapseml_2.12:1.1.3" \
  job.py

The top-level synapseml_2.12 artifact is an aggregate artifact: its POM references the SynapseML module JARs and their runtime dependencies, but the artifact JAR itself does not contain implementation classes. In particular, IsolationForest is in synapseml-core_2.12 and also requires the LinkedIn Isolation Forest dependency. Spark resolves this complete dependency closure when --packages or spark.jars.packages is used.

If a cluster image or pre-installed environment must be manually provisioned, use a Maven-compatible dependency resolver during setup to install the complete runtime dependency closure for the selected coordinate, and verify that the JARs are added to both the driver and executor JVM classpaths (for example via spark.driver.extraClassPath and spark.executor.extraClassPath or standard cluster classpath configuration).

Manual JAR placement is error-prone:

• Merely having a JAR localized or added to Python/YARN/Livy search paths does not ensure that its classes are loaded into the Spark driver's JVM classpath.
• Copying only the aggregate artifact or omitting transitive dependencies will cause ClassNotFoundException or NoClassDefFoundError (often surfaced via Py4J as <class> does not exist in the JVM) even if the Python module imports successfully.

Always pin a single coordinate matching the cluster's Spark and Scala versions.

SBT

Choose the dependency matching your Spark runtime.

Spark 4.1

resolvers += "SynapseML" at "https://mmlspark.blob.core.windows.net/maven"
libraryDependencies +=
  "com.microsoft.azure" % "synapseml_2.13" % "1.1.3-spark4.1"

Spark 4.0

resolvers += "SynapseML" at "https://mmlspark.blob.core.windows.net/maven"
libraryDependencies +=
  "com.microsoft.azure" % "synapseml_2.13" % "1.1.3-spark4.0"

Spark 3.5

resolvers += "SynapseML" at "https://mmlspark.blob.core.windows.net/maven"
libraryDependencies +=
  "com.microsoft.azure" % "synapseml_2.12" % "1.1.3"

Apache Livy and HDInsight

To install SynapseML from within a Jupyter notebook served by Apache Livy, the following Spark 3.5 / Scala 2.12 configure magic can be used. You will need to start a new session after this configure cell is executed.

Excluding certain packages from the library may be necessary due to current issues with Livy 0.5.

%%configure -f
{
    "name": "synapseml",
    "conf": {
        "spark.jars.packages": "com.microsoft.azure:synapseml_2.12:1.1.3",
        "spark.jars.repositories": "https://mmlspark.blob.core.windows.net/maven",
        "spark.jars.excludes": "org.scala-lang:scala-reflect,org.apache.spark:spark-tags_2.12,org.scalactic:scalactic_2.12,org.scalatest:scalatest_2.12,com.fasterxml.jackson.core:jackson-databind"
    }
}

Docker

The easiest way to evaluate SynapseML is via our pre-built Docker container. To do so, run the following command:

docker run -it -p 8888:8888 -e ACCEPT_EULA=yes mcr.microsoft.com/mmlspark/release jupyter notebook

Navigate to http://localhost:8888/ in your web browser to run the sample notebooks. See the documentation for more on Docker use.

To read the EULA for using the docker image, run docker run -it -p 8888:8888 mcr.microsoft.com/mmlspark/release eula

R

To try out SynapseML using the R autogenerated wrappers see our instructions. Note: This feature is still under development and some necessary custom wrappers may be missing.

Building from source

SynapseML has recently transitioned to a new build infrastructure. For detailed developer docs, please see the Developer Readme.

If you are an existing SynapseML developer, you will need to reconfigure your development setup. We now support platform-independent development and better integrate with IntelliJ and SBT. If you encounter issues, please reach out to our support email!

Papers

• Large Scale Intelligent Microservices

• Conditional Image Retrieval

• MMLSpark: Unifying Machine Learning Ecosystems at Massive Scales

• Flexible and Scalable Deep Learning with SynapseML

• Large-Scale Automatic Audiobook Creation

Learn More

• Visit our website.

• Watch our keynote demos at the Spark+AI Summit 2019, the Spark+AI European Summit 2018, the Spark+AI Summit 2018 and SynapseML at the Spark Summit.

• See how SynapseML is used to help endangered species.

• Explore generative adversarial artwork in our collaboration with The MET and MIT.

• Explore our collaboration with Apache Spark on image analysis.

Contributing & feedback

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

See CONTRIBUTING.md for contribution guidelines.

To give feedback and/or report an issue, open a GitHub Issue.

Other relevant projects

• Vowpal Wabbit

• LightGBM

• DMTK: Microsoft Distributed Machine Learning Toolkit

• Recommenders

• JPMML-SparkML plugin for converting SynapseML LightGBM models to PMML

• Microsoft Cognitive Toolkit

Apache®, Apache Spark, and Spark® are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries.