spark-nlp

📢 Spark NLP 6.4.2: BM25 Retrieval, SaT Sentence Detection, and Model Conversion Workflows

Spark NLP 6.4.2 is a feature and reliability release focused on strengthening retrieval, sentence segmentation, model conversion, and release infrastructure. This release introduces BM25Approach and BM25Model for Okapi BM25 lexical retrieval, adds SentenceDetectorSaTModel for transformer-based sentence boundary detection with SaT / Segment any Text models, and expands model conversion examples with new Hugging Face to Spark NLP notebooks for ONNX, OpenVINO, and GGUF workflows.

In addition, this release improves production robustness by properly closing resource streams and file systems during local resource copying, which avoids leak warnings that can affect dependent pipelines. It also updates jsl-llamacpp to 2.0.3.

🔥 Highlights

• BM25Approach + BM25Model: New Okapi BM25 lexical retrieval annotators implemented as a fit-once, query-many Estimator/Model pair. They complement semantic retrieval components such as DocumentSimilarityRanker with a fast, corpus-statistics-based lexical ranking option.
• SentenceDetectorSaTModel: New sentence detector based on SaT / Segment any Text transformer models. It uses ONNX-backed XLM-R SentencePiece models to produce sentence spans from per-token boundary probabilities and supports long documents through overlapping token windows.
• Model conversion notebooks: New end-to-end notebooks demonstrate Hugging Face to Spark NLP conversion flows for ONNX, OpenVINO, and GGUF models.

🚀 New Features & Enhancements

BM25 Lexical Retrieval

Spark NLP 6.4.2 introduces BM25 lexical document retrieval through a two-stage Spark ML design. This implements SPARKNLP-1145 and adds a purely lexical ranking option that complements semantic retrieval components such as DocumentSimilarityRanker.

• BM25Approach scans the tokenized corpus during fit() and learns corpus-level statistics: document count, per-term document frequency, average document length, and inverse document frequency.
• BM25Model reuses those learned statistics at transform time to score each document against a query.

This design is necessary because BM25 scores depend on corpus-level statistics that are only known after fitting over the full dataset. Once fitted, the model can be reused across many queries without rescanning the corpus.

Key capabilities:

• Emits BM25_RANKINGS annotations.
• Adds bm25_score, num_query_terms_matched, query, and doc_len metadata.
• Supports range-validated k1, b, minDocFreq, and caseSensitive parameters.
• Uses the non-negative Lucene / Elasticsearch-style IDF variant.
• Supports setQuery(...) for raw query strings.
• Supports setQueryTokens(...) for pre-analyzed query terms when the corpus pipeline includes normalization, stemming, lemmatization, or other token transformations.
• Makes caseSensitive read-only on fitted BM25Model to avoid corrupting scores after the IDF vocabulary has been built.
• Includes Scala and Python tests for ranking, save/load, exact-score verification, query reuse, parameter propagation, analyzer symmetry, and invalid parameter handling.

from sparknlp.base import DocumentAssembler
from sparknlp.annotator import Tokenizer, StopWordsCleaner, BM25Approach
from pyspark.ml import Pipeline
from pyspark.sql.functions import col, explode

corpus = spark.createDataFrame([
    (1, "Apples are a great source of dietary fiber and vitamin C."),
    (2, "Machine learning uses neural networks and statistical models."),
    (3, "Vitamin C deficiency can affect the immune system."),
], ["id", "text"])

document = DocumentAssembler() \
    .setInputCol("text") \
    .setOutputCol("document")

tokenizer = Tokenizer() \
    .setInputCols(["document"]) \
    .setOutputCol("token")

stop_words = StopWordsCleaner() \
    .setIn

📢 Spark NLP 6.4.1: Late Chunking, Multimodal Embeddings, and Smarter Pretrained Workflows

Spark NLP 6.4.1 is a feature-rich follow-up release that expands retrieval and multimodal capabilities while improving pretrained model loading and production robustness. This release introduces two powerful new annotators: LateChunkEmbeddings for context-aware chunk embeddings and BiEncoderMultimodalEmbeddings for dual-encoder text-image retrieval workflows.

In addition, VectorDBConnector now supports multimodal image indexing, pretrained model resolution becomes smarter with preferred engine selection, and document ingestion workflows gain important reliability fixes for HTML readers and cloud-backed pretrained caches.

For a deeper walkthrough of the major additions in this release, see our Medium post: Spark NLP 6.4.1: Context-Aware Retrieval, Engine Control, and Multimodal Indexing.

🔥 Highlights

• LateChunkEmbeddings: A new annotator that applies the late chunking technique so chunk embeddings are computed from token embeddings generated over the full document context rather than embedding each chunk independently.
• BiEncoderMultimodalEmbeddings: A new multimodal dual-encoder annotator for generating aligned text and image embeddings for retrieval, search, and indexing workflows.
• Multimodal VectorDBConnector: VectorDBConnector now supports indexing image embeddings in addition to text embeddings, enabling multimodal vector database pipelines.
• Smarter .pretrained() resolution: Pretrained model loading now supports a preferred engine parameter and resolves duplicate model names more reliably by filtering first by annotator class.
• Improved production robustness: Fixes for cloud-backed pretrained caches and HTML reader header handling make Spark NLP more reliable in serverless and cloud-storage-based workflows.

🚀 New Features & Enhancements

LateChunkEmbeddings

Spark NLP 6.4.1 introduces LateChunkEmbeddings, a new annotator based on the Late Chunking approach. Unlike traditional chunk embedding approaches that embed each chunk in isolation, this annotator assumes the upstream embedding model has already encoded the full document in a single pass. It then pools token embeddings corresponding to each chunk span, allowing every chunk embedding to retain broader document context.

This is especially valuable for:

• Retrieval-augmented generation (RAG)
• Semantic search over long documents
• Chunk retrieval where local spans depend on earlier context

A follow-up improvement in this same release adds sentenceAwareFiltering, which is enabled by default. This restricts pooling to token embeddings within the chunk’s sentence boundary, reducing noise from overlapping tokens across adjacent sentences.

late_chunk = LateChunkEmbeddings() \
    .setInputCols(["document", "chunk", "token", "embeddings"]) \
    .setOutputCol("chunk_embeddings")

See the LateChunkEmbeddings example notebook for a full walkthrough and practical usage examples. You can also read our Medium article for a broader explanation of why late chunking improves retrieval quality over naive chunk-first embedding pipelines.

BiEncoderMultimodalEmbeddings

This release adds BiEncoderMultimodalEmbeddings, a new annotator for multimodal dual-encoder workflows. It accepts aligned DOCUMENT and IMAGE annotations and emits two embedding outputs:

• <outputCol>_doc_embeddings
• <outputCol>_image_embeddings

This enables use cas

📢 Spark NLP 6.4.0: LLM-Powered Entity Extraction, Expanded Document Readers, and Pipeline Robustness

Spark NLP 6.4.0 is a feature-rich release focused on bringing LLM-powered extraction into standard NLP pipelines, dramatically expanding the document reader ecosystem, and hardening production ingestion workflows. This release introduces LLMEntityExtractor, a new annotator that leverages on-device GGUF LLMs to extract named entities from text using flexible few-shot prompting, no fine-tuning required. We also introduce DocumentTitleSplitter, a semantic chunker for reader-based pipelines, and extend the document reader stack with native support for four new file formats: ODT, EPUB, TSV, and RTF. Finally, reader pipelines gain significant production robustness improvements including graceful error handling for HTML and image fetch failures.

🔥 Highlights

• LLMEntityExtractor: Use any GGUF-based LLM to extract named entities from text with configurable few-shot examples, no model fine-tuning needed. Returns results as standard CHUNK annotations compatible with the rest of the Spark NLP ecosystem.
• DocumentTitleSplitter: A new semantic annotator that splits element-level reader output into coherent chunks using title boundaries, designed to work alongside Reader2Doc in element mode across Word, PDF, HTML, PowerPoint, ODT, and Markdown files.
• Expanded Reader Format Support: ODT, EPUB, TSV, and RTF files are now first-class citizens in the Spark NLP reader stack, joinng the existing support for Word, PDF, HTML, PowerPoint, Markdown, CSV, and more.
• Resilient Reader Pipelines: HTMLReader and Reader2Image now degrade gracefully on unreachable URLs and broken remote images rather than failing the entire Spark job critical for large-scale batch processing and ETL workflows.
• Databricks Unity Catalog Volume support: Documentation and instructions added for loading pretrained models from Databricks Unity Catalog Volumes.

🚀 New Features & Enhancements

LLMEntityExtractor

Teams that need flexible entity extraction without model fine-tuning can now use LLMEntityExtractor to leverage on-device GGUF models (via AutoGGUFModel) for NER-style extraction. The annotator uses structured prompts (optionally enhanced with few-shot examples) to identify and extract entities from input text, returning them as standard CHUNK annotations.

Key capabilities:

• Works with any GGUF-format LLM supported by AutoGGUFModel
• Supports configurable few-shot examples via FewShotExample objects
• Reasoning budget is set to 0 by default to minimize latency
• Includes partial JSON recovery for robustness against malformed LLM outputs
• Fully integrated into Python and Scala APIs, including .pretrained() and .loadSavedModel()

llm_extractor = LLMEntityExtractor.pretrained() \
    .setInputCols(["document"]) \
    .setOutputCol("entities")

See the LLMEntityExtractor example notebook for full usage including few-shot configuration.

DocumentTitleSplitter

Reader pipelines that use Reader2Doc in element mode produce structured output with elementType metadata (e.g., Title, NarrativeText). DocumentTitleSplitter closes a key gap by turning this element sequence into semantic, title-bounded chunks that downstream annotators like sentence embedders and LLMs can consume coherently.

Key parameters:

• setSplitOnPageBreak(bool): Also enforce chunk boundaries on page breaks
• setJoinDelimiter(str): Delimiter used when joining narrative text within a chunk
• setMaxChunkLength(int): Overflow splitting for oversized title sections

Supported document types include Word, PDF, HTML, PowerPoint, ODT, and Markdown any format for which Reader2Doc extracts title metadata

📢 Spark NLP 6.3.3: ModernBERT Embeddings, Vector DB Integration, and Layout-Aware Document Processing

Spark NLP 6.3.3 is a feature-packed release aimed at practitioners building modern NLP and multimodal document pipelines. This release introduces ModernBertEmbeddings for dramatically faster and more memory-efficient text embeddings, VectorDBConnector to close the gap between embedding pipelines and vector search infrastructure.

Moreover, we introduce a new suite of document-understanding annotators: LayoutAlignerForVision and LayoutAlignerForText which together enable coherent end-to-end multimodal pipelines over complex documents like PDFs and PowerPoint files. To gain an in-depth walkthrough on how to build use your own pipelines and documents, please see our Medium blog post at Efficient Document Ingestion with Layout Aware Annotators: A Case Study on Mixed-Type Documents.

In addition, a new MultiColumnAssembler can merge multiple annotation column into one and LightPipeline also gains metadata column support for more powerful batch inference workflows.

🔥 Highlights

• ModernBertEmbeddings: A state-of-the-art encoder that is 8x faster and uses 5x less memory than traditional BERT, with native support for sequences up to 8192 tokens which is ideal for long-document NLP tasks.
• VectorDBConnector: Bridges Spark NLP embedding pipelines with external vector databases (initially Pinecone), so teams building semantic search and RAG systems no longer need custom glue code to store embeddings.
• LayoutAlignerForVision and LayoutAlignerForText: New annotators for multimodal document pipelines that spatially align text and images from complex documents, giving downstream Vision-Language Models (VLMs) the coherent context they need to produce better results.
• MultiColumnAssembler: Closes a common pipeline gap when ReaderAssembler splits document content across multiple columns (text, table, image captions). This annotator merges them back into a single column that downstream annotators like AutoGGUFVisionModel expect.
• Enhanced LightPipeline with metadata column support for richer, context-aware inference workflows.

🚀 New Features & Enhancements

ModernBertEmbeddings

Teams working with long documents, code, or large-scale embedding workloads will benefit from ModernBertEmbeddings, which brings the latest generation of bidirectional encoder models to Spark NLP. Based on the paper Smarter, Better, Faster, Longer, ModernBERT was trained on 2 trillion tokens with a native sequence length of up to 8,192 tokens (eight times the limit of classic BERT) enabling faster, cheaper embeddings for longer sequences without truncation.

• Default pretrained model: "modernbert-base" (English)
• 768-dimensional token-level WORD_EMBEDDINGS output

embeddings = ModernBertEmbeddings.pretrained() \
    .setInputCols(["document", "token"]) \
    .setOutputCol("modernbert_embeddings")

See the ModernBertEmbeddings notebook for extended examples, including how to import custom HuggingFace ModernBERT models via ONNX.

VectorDBConnector

For teams building semantic search, retrieval-augmented generation (RAG), or similarity-based recommendation systems, manually bridging Spark NLP with a vector database has historically required custom integration code. VectorDBConnector eliminates this gap by letting you store embeddings from any Spark NLP embedding annotator directly into a vector database as part of the pipeline. It initially supports Pinecone, with more providers planned