infinity

michaelfeilMIT活跃维护

面向文本嵌入、重排、CLIP、CLAP 和 ColPali 模型的高吞吐低延迟推理服务引擎，适合把向量化和重排模型部署成线上 API。

Infinity is a high-throughput, low-latency serving engine for text-embeddings, reranking models, clip, clap and colpali

0.0.80发布于 2026-11-09 03:45 UTC2,984star222fork同步于 2026-11-10 05:41 UTC

推理引擎文本嵌入重排模型向量化服务低延迟

对比 GitHub 官网 bert-embeddings llm text-embeddings

怎么上手

由 README 提炼，执行前请确认

原文

docker run --rm --gpus all -p 8000:8000 -v infinity-cache:/data ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.0 --model-id sentence-transformers/all-MiniLM-L6-v2 --torch-device cuda --dtype bfloat16 --hostname 0.0.0.0 --port 8000

硬件要求按 README 说明

需要 NVIDIA GPU，算力 SM80（Ampere）及以上，并安装 NVIDIA Container Toolkit；README 未说明 CPU 运行方式。每张 GPU 必须能装下完整模型，MoE 模型需包含全部专家。

需要 CUDA有 Docker 镜像

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

没人工点评？写下你的经验

Rust 写的推理服务引擎，把 embedding、rerank、分类等模型统一成 REST API

面向文本嵌入、重排、CLIP、CLAP 和 ColPali 模型的高吞吐低延迟推理服务引擎，适合把向量化和重排模型部署成线上 API。

核心能力

用 Rust 和 LibTorch 原生实现 v3 推理服务
通过 /v1/embeddings 提供文本与多模态向量
支持 /rerank 重排与 /v1/classify 分类打分
提供 SPLADE 稀疏向量与 token 级向量检索
每张 GPU 放一份完整模型副本并行服务

容易踩的坑

v3 是 Rust 原生服务，与 infinity-emb Python 包和 infinity_emb v2 命令不是同一个东西；模型需从 Hugging Face 拉取并缓存到挂载的 /data 卷；v3 采用自定义的 Infinity Restricted MIT 许可，并非标准 MIT。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本0.0.80

全部 release

v3.0.2-beta.3Infinity v3.0.2-beta.3pre-release2026-11-09 · 2026-11-09 07:07 UTC
查看更新说明
Infinity V3 3.0.2-beta.3 updates native CUDA inference and restores compatibility with legacy rerank requests.
- Rerank requests accept documents as an alias for texts (#906).
- Dynamic quantization is selected per model with --dynamic-quantization none|fp8|mxfp8|nvfp4 and --dynamic-quantization-scope auto|mlp|all (#910). Quantization remains disabled by default.
- Compatible Blackwell BF16 decoder MLPs can use native MXFP8 or opt-in NVFP4. FP8 conversion, projection scaling, activation, and RoPE paths have further native optimizations (#905 and related commits).
- Shared-prefix attention uses paged FA4 for supported layouts (#914). GDN convolution and provider packing are fused, and eligible cached GDN suffix prefill is selected automatically (#911, #912).
Migration from earlier betas:
- Replace --enable-fp8-dynamic and the retired FP8/MXFP8/NVFP4 environment selectors with the new per-model options. Old selectors are rejected, including when set to 0 or false. /info now reports dynamic_quantization and dynamic_quantization_scope. See the migration table.
- The INFINITY_TORCH_GDN_SUFFIX_PREFILL opt-out is removed. Cached suffix prefill is automatic when its prefix state and provider support are available. BF16 arithmetic can change probabilities and selected answers; validate outputs for your workload.
Images for linux/amd64:
- ghcr.io/michaelfeil/infinity_v3:cpu-3.0.2-beta.3
- ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.2-beta.3
Both images pass version/startup and MiniLM embedding inference smoke checks before publication. The NVIDIA image's CI inference check runs on CPU; GPU hardware execution is not part of that check. This prerelease leaves stable image and documentation aliases unchanged.

Changes since beta.2 · V3 licensing and retained notices
0.0.80Infinity v1 0.0.80 (Python 3.11+)最新正式版2026-11-09 · 2026-11-09 03:45 UTC
The legacy Python v1 server and client now require Python 3.11 or newer. Server CI covers Python 3.11, 3.12, and 3.13 on Linux and Windows. The previous 0.0.79 release remains available for older Python environments.

Changes:
- Move containers to Ubuntu 24.04 and Python 3.12; the NVIDIA image uses CUDA 12.9.1.
- Refresh CI tooling and compatible inference dependencies, including NumPy 2 and Torch 2.8.
- Preserve checkpoint pooling, existing ColPali models, and AMD/BetterTransformer support through compatible dependency bounds. Match ONNX Runtime GPU 1.23 and TensorRT 10.9 to CUDA 12.
- Fix ONNX loading through symlinked local model directories, including Hugging Face snapshot file symlinks.
- Isolate client-generation tooling and retain the Python client's existing HTTPX compatibility.
Release automation publishes:
- PyPI: infinity_emb==0.0.80 and infinity_client==0.0.80.
- NVIDIA: michaelf34/infinity:0.0.80 for linux/amd64 (CUDA-enabled PyTorch) and linux/arm64 (CPU-only PyTorch).
- CPU: michaelf34/infinity:0.0.80-cpu for linux/amd64.
The existing 0.0.79 version tags and images remain unchanged. The ARM64 variant continues to use CPU-only PyTorch and does not provide ARM64 CUDA acceleration. CI validated CPU/NVIDIA amd64 container builds; GPU hardware execution was not exercised in CI.

This release belongs to the legacy Python v1 line; v3 releases remain separate.

Full changelog: https://github.com/michaelfeil/infinity/compare/0.0.79...0.0.80
0.0.79Infinity v1 maintenance release 0.0.792026-11-08 · 2026-11-08 20:58 UTC
查看更新说明
Infinity v1 / legacy Python maintenance release.
- Allow HTTPX 0.28 in infinity_client, with proxy examples compatible with HTTPX 0.20–0.28.
- Upgrade Typer to the 0.20 series and allow Click 8.x, fixing CLI startup with Click 8.3 (#650, #651, #907).
- Preserve the HTTPX dependency constraint when regenerating the Python client.
- Keep Python lint and test jobs on Ubuntu 24.04 for the existing Python support range.
- Fix the NVIDIA ARM64 image build attempting to install an x86-only optional FlashAttention wheel.
Python support remains unchanged: server Python >=3.9,<3.14; client Python >=3.8,<4.0.

Install:
```
pip install 'infinity_emb[all]==0.0.79'
pip install 'infinity_client==0.0.79'
```
PyPI packages and Docker images publish asynchronously through the release workflows.

Full changelog
v3.0.2-beta.1Infinity v3.0.2-beta.1pre-release2026-10-31 · 2026-10-31 20:19 UTC
查看更新说明
Infinity V3 3.0.2-beta.1 improves shared-prefix decision inference and consolidates the native backend on LibTorch.
- OneJev block-attention requests execute as whole groups and can share a backend batch within configured limits. Oversized groups return HTTP 422 before inference.
- Shared prefix K/V pages, bounded temporary K/V copies and compact last-token readout reduce fanout memory use. Compatible Qwen tokenizers reuse the shared prefix encoding within each request.
- Eligible cached GDN suffixes use tensor-core prefill by default. This matches the previous INFINITY_TORCH_GDN_SUFFIX_PREFILL=1 setting; use =0 for recurrent suffix execution. BF16 normalization and chunk arithmetic can change probabilities and selected answers relative to the previous default.
- Optional service-tier queue token limits shed excess queued work. Pixtral image preprocessing is faster.
- The Candle backend has been removed. Both CPU and NVIDIA images now use native LibTorch.
Large fanouts still require suitable compute, expanded-token and question-count limits. See the batch configuration documentation.

CPU/H100 native regressions, GDN reference checks and tokenizer parity checks passed during integration. The default GDN setting matched explicit opt-in exactly in the Elephant fanout check.

Container builds are triggered for linux/amd64:
- ghcr.io/michaelfeil/infinity_v3:cpu-3.0.2-beta.1
- ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.2-beta.1
Images become available after their build and inference smoke checks finish. This prerelease leaves stable latest aliases unchanged.

Changes since v3.0.1 · V3 licensing and retained notices
v3.0.1-beta.3Infinity v3.0.1-beta.3pre-release2026-10-28 · 2026-10-28 16:35 UTC
查看更新说明
Infinity V3 3.0.1-beta.3 includes configurable service-tier response headers and optional priority billing-token scaling.
- Set INFINITY_BILLING_HEADERS=x-baseten to emit x-baseten-input-tokens and x-baseten-service-tier (default or priority). Zero priority queue bonus reports default.
- Set INFINITY_EXPERIMENTAL_MULTIPLIER=1.2 to scale only effective-priority billing-token counts. It is unset by default. Decimal arithmetic is exact and results round up to integer tokens; actual token usage is unchanged.
- Also includes D3 Ultra checkpoint support and embedding tensor wire-contract clarifications since beta.2.
Images: ghcr.io/michaelfeil/infinity_v3:cpu-3.0.1-beta.3 and ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.1-beta.3 (linux/amd64). Prerelease tags leave stable latest image aliases unchanged.

CPU images use Candle; the NVIDIA images use native LibTorch. Model support differs by backend. See V3 licensing and the scoped package notices.

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

共 0 条

还没有人点评。第一个说出真实体验的人最有价值。

README 原文

未翻译

在 GitHub 打开

Infinity v3 ♾️

Infinity serves embeddings, reranking, classification, and typed decision models through a REST API. The native v3 server is written in Rust and uses LibTorch, the C++ interface of PyTorch, for model inference.

Looking for Python / v2? See the Python server README and v2 documentation. The infinity-emb Python package and infinity_emb v2 command belong to that server.

History

2026/10: Native Rust/LibTorch v3 server for embeddings, reranking, classification and typed decisions.
2025/07: Blackwell support in the Python server.
2024: Python / v2 gained multi-model and multimodal serving, Python clients, and broader GPU, CPU, ONNX and TensorRT deployment options.
2023/10: Initial Infinity release.

See the Python / v2 README for the full earlier updates.

Run with Docker

Use an NVIDIA GPU with compute capability SM80 (Ampere) or newer and NVIDIA Container Toolkit:

docker run --rm --gpus all -p 8000:8000 -v infinity-cache:/data \
  ghcr.io/michaelfeil/infinity_v3:nvidia-3.0.0 \
  --model-id sentence-transformers/all-MiniLM-L6-v2 \
  --torch-device cuda --dtype bfloat16 \
  --hostname 0.0.0.0 --port 8000

The image starts infinity_v3 directly. All visible CUDA GPUs are used by default, with a complete model replica on each GPU. Pooling is selected from the checkpoint metadata. Pin a published version or image digest from the container registry for reproducible deployments.

Send an embedding request:

curl http://localhost:8000/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model":"sentence-transformers/all-MiniLM-L6-v2","input":["A document about semantic search."]}'

Open localhost:8000/docs for the running server's interactive API. /health reports readiness after loading and warmup.

Models and tasks

The supported model catalog groups named checkpoints by task, with configuration, input requirements, support basis, and model-license notes:

Dense text and multimodal embeddings through /v1/embeddings.
Token-vector embeddings for late-interaction retrieval.
Reranking through /rerank.
Sequence classification and reward scoring through /v1/classify.
Token classification and named-entity recognition through /predict_tokens.
Typed decisions through /v1/systemone.
SPLADE sparse embeddings through /embed_sparse.

Support depends on the checkpoint's architecture and trained task head. Each GPU must fit the full model, including all experts for MoE checkpoints.

Documentation

Licensing

Covered original v3 contributions use the Infinity Restricted MIT License 1.0, with retained third-party and earlier grants. This is a custom source-available license; see the full v3 license for its terms. The Python packages retain the MIT License. Model weights and dependencies have their own licenses.