mteb

2.27.4 (2026-11-09)

Fix

• fix: keep top_k results per query when ignore_identical_ids is set (#5844)

• fix: keep top_k results per query when ignore_identical_ids is set

With ignore_identical_ids, the query's own document is dropped only at evaluation time, after search has already kept exactly top_k documents. A query that retrieved itself (usual for ArguAna, Quora and similar tasks) was scored on top_k - 1 documents, which lowered every metric at the largest cutoff.

Search one extra document for these tasks, and after dropping the identical id trim each query back to max(k_values), breaking ties like trec_eval. Tasks without ignore_identical_ids are unchanged.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

• update test

• simplify comment

Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Roman Solomatin <[email protected]> (bde81c7)

• fix: video frame timing (#5652)

• fix: OmniVinci and Gemini get video files that play at the real frame spacing

Both receive the sampled frames as a video file: OmniVinci's was written at a fixed 24 fps (32 frames of any clip spanned 1.3 s, and its time embeddings read these times) and Gemini's at 1 fps (a long clip capped at 32 frames became a 32 s video). The video collators can now return each clip's frame timing (video_metadata=True), and both files are written at the frames' real spacing, as when the models get the original video.

• fix: Qwen3-Omni, GVE, Qwen3-VL-Embedding and BidirLM-Omni get real frame timing

• Qwen3-Omni: its processor assumed 1 fps while mteb samples at 2 fps; it now passes the sampling rate, as qwen-omni-utils does.
• GVE, Qwen3-VL-Embedding, BidirLM-Omni: their processors assume 24 fps without video_metadata, so a 4-minute clip was timed as under 3 s; each clip's timing is now passed, as their official code does via qwen-vl-utils.

The Qwen3-VL reranker is left as is (sentence-transformers' cross-encoder does not forward video_metadata), and so is ColQwen2.5-Omni (colpali-engine has the same mismatch).

• Review feedback: real per-video rate for Qwen-Omni, VideoMetadata type, drop the Gemini change

• Qwen-Omni gets each video's real frame rate (the rate frames were sampled at is not self.fps when max_frames caps a long clip, and is unset with num_frames). Its processors take one rate per call, so video is encoded one clip at a time.
• The collators' frame timing is a VideoMetadata TypedDict (fields of transformers' VideoMetadata), returned in a VideoInputItem.
• Gemini is back to main: the API resamples to at most 32 frames itself, and real-length files can exceed its 120 s video limit.
• Release notes are left untouched.

• Qwen-Omni: compute the frame rate as qwen-omni-utils does

sample_fps = sampled frames / source frames * source fps, instead of the frames' spacing.

• Apply review suggestion: VideoInput.video is a tensor or a VideoInputItem

• Always return frame timing as batch["video_metadata"]

Review feedback: the video collators always add each video's timing (VideoMetadata) next to the frames, which stay a plain tensor, instead of an opt-in flag that changes the frames' format. Wrappers that don't use it are unchanged; GVE, OmniVinci and Qwen-Omni read the new key, and Qwen3-VL-Embedding and BidirLM-Omni join it with the frames for sentence-transformers.

• Review feedback: resample_video always returns frame timing, no flags

• FramesCollator.resample_video and _sample_video return (frames, timing) instead of a return_metadata flag or modifying the row in place.
• InstructSentenceTransformerModel always passes video as sentence-transformers' {array, video_metadata} format (only Qwen3-VL-Embedding uses it with video), so its video_metadata flag is gone.
• VideoMetadata.

2.27.3 (2026-11-09)

Fix

• fix: disable KV cache for stella_en_1.5B_v5 so it runs on current transformers (#5850)

• fix: disable KV cache for stella_en_1.5B_v5 so it runs on current transformers

The remote modeling_qwen.py at the pinned revision calls DynamicCache.get_usable_length(), which was removed in transformers 4.54. With the mteb entry's dtype kwarg (transformers >= 4.56), the model loads but fails on the first forward pass, so it could not run on any version.

Encoding never uses the KV cache, so pass config_kwargs={"use_cache": False} to skip that code path, as already done for gte-Qwen2-7B-instruct (#4935). Outputs are unchanged: on transformers 4.53.3 hidden states with the cache on and off are identical, and match transformers 4.57.6 with the cache off.

Fixes #5832

• set req group

Co-authored-by: Aarush Shetty <[email protected]> Co-authored-by: Roman Solomatin <[email protected]> (34207fc)

Unknown

• model: add BrunoHays/embeddinggemma-2-nullspace-audio (#5775)

• model: add BrunoHays/embeddinggemma-2-nullspace-audio

• chore(models): rename file to embeddinggemma2_nullspace.py

• refactor(models): inject null-space audio features inside model.forward

Co-authored-by: Cursor Agent <[email protected]> (e47213f)

2.27.2 (2026-11-09)

Fix

• fix: add GoogleSVQ v2 with one text per sentence (#5790)

• fix: add GoogleSVQ v2 with one text per sentence

• normalize SVQ transcriptions before grouping sentences

• regenerate SVQ v2 stats after normalization

• Use the same folding as the near-duplicate analysis for SVQ sentences

• Build GoogleSVQ v2 splits in self.dataset (010d62f)

Unknown

• model: add Reza2kn/RaahNaame-Koochik-v1 (Persian embedding model) (#5851)

• model: add Reza2kn/RaahNaame-300m

• model: use SentenceTransformerEncoderWrapper and google_gemini import (current main)

• model: rename to Reza2kn/RaahNaame-Koochik-v1

• model: drop non-paper citation (review) (9242b36)

• model: add opensearch-neural-embed-v1 and opensearch-neural-sparse-encoding-asym-v1 (#5852)

• model: add opensearch-neural-embed-v1 and opensearch-neural-sparse-encoding-asym-v1

• fix: avoid importing torch at module import in opensearch_neural_embed_models (50c3917)

2.27.1 (2026-11-05)

Fix

• fix: Clarify documentation on output_dtypes and update in ModelMeta (#5723)

• Clarify documentation on output_dtypes and update in ModelMeta

• change from review (1842b7e)

Unknown

• model: Add mzahran1/arabic-english-e5-small-retrieval and omarelshehy/Arabic-Retrieval-v1.0 (#5846)

Co-authored-by: Mahmoud Zahran <[email protected]> (b9ccd7a)