登录

model_server

openvinotoolkitApache-2.0活跃维护

把 OpenVINO 优化过的模型部署成可扩展推理服务的服务器，支持云端和边缘、Kubernetes 与 DAG 流水线，方便生产环境上线模型。

A scalable inference server for models optimized with OpenVINO™

v2026.4.1发布于 2026-10-06 21:12 UTC956star +1 / 7 天294fork同步于 2026-11-10 11:51 UTC
推理服务模型部署OpenVINO可扩展边缘推理
对比 GitHub 官网aiclouddagdeep-learningedge

怎么上手

由 README 提炼，执行前请确认
原文 
docker run --rm -p 8000:8000 --user $(id -u):$(id -g) -v ${HOME}/models:/models:rw openvino/model_server:latest --source_model OpenVINO/Qwen3-4B-int4-ov --model_repository_path /models --rest_port 8000
硬件要求按 README 说明

支持 Intel CPU（含 Xeon）、Intel 集成和独立 GPU 以及 NPU；GPU 加速需用 latest-gpu 镜像并暴露 /dev/dri 设备；README 未说明显存要求。

有 Docker 镜像
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

用 OpenVINO 优化的 C++ 推理服务器，同时服务生成式 AI 和经典深度学习模型

把 OpenVINO 优化过的模型部署成可扩展推理服务的服务器，支持云端和边缘、Kubernetes 与 DAG 流水线，方便生产环境上线模型。

核心能力

  • 提供 OpenAI 兼容 API，支持文本生成、嵌入、图像生成和音频
  • 提供 KServe API 服务经典模型推理
  • 支持连续批处理、流式输出、结构化输出和投机解码
  • 支持 TensorFlow、ONNX、PaddlePaddle、OpenVINO IR 和 GGUF 格式
  • 模型仓库支持本地、S3、GCS、Azure Blob 和 HuggingFace Hub

容易踩的坑

首次运行会从 HuggingFace 自动下载模型到挂载目录，需保证网络和磁盘空间；GPU 加速必须换用 latest-gpu 镜像并额外传入 --device /dev/dri 和 --group-add 参数；经典模型示例需先手动下载 resnet50.bin 和 resnet50.xml。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v2026.4.1
全部 release
  1. v2026.4.1OpenVINO Model Server 2026.4.1最新正式版2026-10-06 · 2026-10-06 21:12 UTC

    This is a hotfix release including bug fixes in the server component and OpenVINO Runtime.

    Improvements included:

    Deprecation notice

    Starting from next release the following capabilities are planned to be dropped:

    • packages targeted on Ubuntu22 will not be published. Best effort support for building will be supported until 2027.0. There is planned adding Ubuntu26 packages.
    • MediaPipe calculators using TensorFlow dependency will be removed from OVMS. In result demos iris_tracking and holistic will be removed. Other calculators with improved performance will be added instead.

    You can use OpenVINO Model Server public Docker images based on Ubuntu via the following command:

    • docker pull openvino/model_server:2026.4.1 - CPU device support with image based on Ubuntu 24.04
    • docker pull openvino/model_server:2026.4.1-gpu - GPU, NPU and CPU device support with image based on Ubuntu 24.04

    or use provided binary packages. Only packages with suffix _python_on have support for python.

    There is also additional distribution channel via https://storage.openvinotoolkit.org/repositories/openvino_model_server/packages/2026.4.1/

  2. v2026.4.0OpenVINO Model Server 2026.4.02026-09-17 · 2026-09-17 13:48 UTC
    查看更新说明

    New models highlights:

    Added support for Qwen3.8-27B model

    Improved accuracy for Gemma-4-26B-a4B on GPU accelerators

    Added support for Muse-Glimmer-30B model including native function calling

    Added support for Kokoro model in speech generation pipeline with execution on NPU accelerator

    New capabilities:

    Introduced preview support for idle model management, enabling models to be automatically unloaded to free RAM when not in use.

    Added option to relax input count validation in KServe API --disable_input_count_validation

    Performance:

    Added preview support for MTP and DFlash

    Added support for Eagle-3 speculative decoding, including tree drafting and chain drafting

    Fixes:

    Resolved issues in NPU execution for Qwen3-Embedding and Qwen3 when context length exceeds 8k tokens. Added configuration parameter --max_length for context length configuration to reduce memory usage in NPU

    Fixed problem in structured output and handling "#" character

    Security related improvements

    Internal changes and improvements:

    Updated Azure SDK and removed boost dependency from the Linux build

    You can use OpenVINO Model Server public Docker images based on Ubuntu via the following command:

    • docker pull openvino/model_server:2026.4.0 - CPU device support with image based on Ubuntu 24.04
    • docker pull openvino/model_server:2026.4.0-gpu - GPU, NPU and CPU device support with image based on Ubuntu 24.04

    or use provided binary packages. Only packages with suffix _python_on have support for python.

    There is also additional distribution channel via https://storage.openvinotoolkit.org/repositories/openvino_model_server/packages/2026.4.0/

  3. v2026.3.1OpenVINO Model Server 2026.3.12026-08-31 · 2026-08-31 13:21 UTC
    查看更新说明

    This is a hotfix release including minor changes in the server component and a few improvements from OpenVINO Runtime.

    Improvements included:

    • Corrections in input validation for KServer and audio endpoints
    • Fixed structured output generation including # char in the content
    • Addressed security vulnerabilities
    • Updated OpenVINO Runtime to 2026.3.1 version
    • Kokoro model can be used with NPU accelerator
    • Gemma4 model has accuracy improvements

    Known issues:

    • Model Qwen3.6-27B is supported only when exported via optimum-intel with OpenVINO 2026.3 version. The default version in https://huggingface.co/OpenVINO/Qwen3.8-27B-int4-ov fails to load unless branch 2026.3.1 is used. The latest weekly package drop and docker image openvino/model_server:weekly don't have such limitation
    • Model Muse-Glimmer-30B is supported without native tool parsing. Tool parser has been added to 2026.4.0. The latest weekly package drop and docker image openvino/model_server:weekly don't have such limitation
    • OMNI models are supported as a preview feature with limitations - MOE models are not supported yet, concurrency is not possible - requests will be queued in processing.
    • NPU execution on LLM has a limit on max prompt parameter 8k tokens. Model will fail to load with higher parameter. It is expected to be fixed in next release.
    • NPU plugin for Qwen3-embeddings and Qwen4-reranking fails to load the model when allowed context exceeds 8k tokens. In order to user such models, the workaround is to edit config.json in the model folder and set "max_position_embeddings" to be below 8k. It is expected to be fixed in next release.

    You can use OpenVINO Model Server public Docker images based on Ubuntu via the following command:

    • docker pull openvino/model_server:2026.3.1 - CPU device support with image based on Ubuntu 24.04
    • docker pull openvino/model_server:2026.3.1-gpu - GPU, NPU and CPU device support with image based on Ubuntu 24.04

    or use provided binary packages. Only packages with suffix _python_on have support for python.

    There is also additional distribution channel via https://storage.openvinotoolkit.org/repositories/openvino_model_server/packages/2026.3.1/

  4. v2026.3OpenVINO Model Server 2026.3.02026-08-04 · 2026-08-04 22:48 UTC
    查看更新说明

    New features

    Improved performance, accuracy, memory consumption, and stability, with particular focus on new models featuring linear attention architectures like qwen3.6.

    Added new tool parsers for LFM2.5 and MiniCPM-V5 models. They can be used in agentic scenarios with native function calling.

    Added support for the Jinja chat template engine as an alternative to Minja for Vision Language Models (VLMs). By default, ovms package with python integration use Jinja while the package without python can use Minja. Jinja provides support for wider set of chat templates, but most models are supported by Minja. Current exceptions are lfm2, miniCPMv5 and Qwen3-Coder

    Introduced automatic detection of key runtime parameters for generative models, including model task category, target device, and tool parsers, reducing required configuration and minimizing deployment errors. Default target device is determined based on the priority: dGPU if present, iGPU if present, CPU. If there are several discrete GPUs, the one with more free VRAM will be selected.

    Simplified deployment of classic models by enabling direct deployment without versioning by pointing ‘--model_path’ to a model file, in addition to a folder with model versions.

    Extended OVMS CLI support for easier configuration and deployment of local models, including the ability to add classic models to multi-model config file alongside generative models.

    Preview support for Kokoro model in OpenAI API ‘/audio/speech’ endpoint, including multi-language support via the optional eSpeek component.

    Extended support for Automatic Speech Recognition (ASR) models beyond the Whisper model family, enabling more generic audio endpoint support for models such as Qwen-ASR.

    Preview support for Qwen3-OMNI models. This pipeline exposes chat/completions and /responses API. On input, is accepted text, images and audio. Model can generate text and audio. Both unary and streaming responses are covered.

    Enabled /v1 REST version for generative endpoints. It can be used just like v3 which remains for compatiblities reasons with previous version. v1 version makes easier integration with some AI frameworks assuming v1 for OpenAI endpoints.

    Added option --verbose_response which makes chat/completions endpoint sending raw model prompts and raw model response. It can be used for debugging purposes or testing. Verbose response is compatible with llama.cpp.

    Bug fixes

    Addressed known security vulnerabilities

    Discontinued

    TensorFlow Server API is now discontinued. KServe API is addressing the same use cases but has more capabilities and more efficient communication pattern.

    Known issues

    • Gemma4 models are supported only on CPU. GPU execution might be not reliable yet. Resolution is expected soon.
    • Kokoro is supported on CPU only. GPU and NPU execution is to be enabled soon.
    • OMNI models are supported as a preview feature with limitations - MOE models are not supported yet, concurrency is not possible - requests will be queued in processing.
    • NPU execution on LLM has a limit on max prompt parameter 8k tokens. Model will fail to load with higher parameter. It is expected to be fixed soon.
    • NPU plugin for Qwen3-embeddings and Qwen4-reranking fails to load the model when allowed context exceeds 8k tokens. In order to user such models, the workaround is to edit config.json in the model folder and set "max_position_embeddings" to be below 8k. It is expected to be fixed soon.
  5. v2026.2.1OpenVINO Model Server 2026.2.12026-06-19 · 2026-06-19 14:31 UTC
    查看更新说明

    This is a minor release focused on bug fixes and improvements related to memory management.

    The following changes are included:

    • updated OpenVINO Runtime to version 2026.2.1
    • updated GPU driver in the docker image with ubuntu24 base image
    • updated NPU driver in the docker image with ubuntu24 base image
    • improvements in input validation for KServer and TFS API
    • added new configuration parameters for LLM and VLM models --cache_interval_multiplier

    New parameter cache_interval_multiplier is relevant only for model models with linear attention like Qwen3.6-35B-A3B. It defines how prefix caching algorithm is managing KV cache allocations. The default value 8 is optimized for short context. While using long prompts like over 20k tokens, it is recommended to increase the value to reduce memory consumption. Here is and example for deploying Qwen3.6-35B-A3B model

    ovms --model_repository_path c:\models --source_model OpenVINO/Qwen3.6-35B-A3B-int4-ov ^
--task text_generation --target_device GPU --tool_parser qwen3coder --reasoning_parser qwen3 ^
--rest_port 8000 --cache_interval_multiplier 64

    You can use OpenVINO Model Server public Docker images based on Ubuntu via the following command:

    • docker pull openvino/model_server:2026.2.1 - CPU device support with image based on Ubuntu 24.04
    • docker pull openvino/model_server:2026.2.1-gpu - GPU, NPU and CPU device support with image based on Ubuntu 24.04

    or use provided binary packages. Only packages with suffix _python_on have support for python.

    There is also additional distribution channel via https://storage.openvinotoolkit.org/repositories/openvino_model_server/packages/2026.2.1/

    Check the instructions how to install the binary package. The prebuilt image is available also on RedHat Ecosystem Catalog

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    OpenVINO™ Model Server

    High-performance model serving for Generative AI and classic deep learning — powered by OpenVINO and optimized for Intel hardware.

    Apache License Docker Pulls GitHub Release Platform

    What is OVMS?

    OpenVINO Model Server (OVMS) is a production-grade, C++ inference server that exposes ML models over standard network APIs. It serves both Generative AI, Agentic workloads (LLMs, VLMs, image generation, audio) and classic deep learning models (object detection, classification, OCR, and more).

    • OpenAI-compatible API for text generation, embeddings, image generation, and audio
    • KServe APIs for classic model inference
    • Runs anywhere — Docker, bare metal, Kubernetes/OpenShift, Windows
    • Intel-optimized — CPU, GPU, NPU acceleration via OpenVINO

    OVMS diagram

    Quick Start

    Serve an LLM with OpenAI-compatible API

    On Linux (Docker):

    mkdir -p ${HOME}/models
# Model is downloaded automatically from HuggingFace to models folder
docker run --rm -p 8000:8000 \
  --user $(id -u):$(id -g) -v ${HOME}/models:/models:rw \
  openvino/model_server:latest \
  --source_model OpenVINO/Qwen3-4B-int4-ov \
  --model_repository_path /models \
  --rest_port 8000

    For GPU acceleration, use the latest-gpu image tag and pass --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1) to expose the Intel GPU device.

    On Windows (binary package):

    mkdir c:\models
ovms.exe --source_model OpenVINO/Qwen3-4B-int4-ov --model_repository_path c:\models --rest_port 8000

    Query the model:

    pip install openai

    from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
stream = client.chat.completions.create(
    model="OpenVINO/Qwen3-4B-int4-ov",
    messages=[{"role": "user", "content": "What are the 3 main tourist attractions in Paris?"}],
    stream=True,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}}
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

    LLM QuickStart

    Serve a Classic Model with KServe API

    Download the model:

    curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.bin -O
curl -L https://huggingface.co/OpenVINO/resnet50-int8-ov/resolve/main/resnet50.xml -O

    On Linux (Docker):

    docker run --rm -d -u $(id -u) -v ${PWD}:/models -p 9000:9000 \
  openvino/model_server:latest \
  --model_name resnet --model_path /models/resnet50.xml \
  --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" \
  --port 9000

    For GPU acceleration, use the latest-gpu image tag and pass --device /dev/dri --group-add $(stat -c '%g' /dev/dri/render* | head -n1) to expose the Intel GPU device.

    Windows (binary package):

    ovms --model_name resnet --model_path resnet50.xml --mean "[123.675,116.28,103.53]" --scale "[58.395,57.12,57.375]" --layout "NHWC:NCHW" --port 9000

    Run inference with a sample client

    pip install numpy tritonclient[grpc]
curl -L -o image.jpeg https://github.com/openvinotoolkit/model_server/blob/main/demos/common/static/images/bee.jpeg?raw=true

    import numpy as np
import tritonclient.grpc as grpcclient
with open("image.jpeg", "rb") as f:
    image_bytes = f.read()
client = grpcclient.InferenceServerClient(url="localhost:9000")
inputs = [grpcclient.InferInput("image", [1], "BYTES")]
inputs[0].set_data_from_numpy(np.array([image_bytes], dtype=object))
outputs = [grpcclient.InferRequestedOutput("output")]
result = client.infer(model_name="resnet", inputs=inputs, outputs=outputs)
output = result.as_numpy("output")  # (1, 1000) FP32
print("Top-1 class index:", int(np.argmax(output[0])))

    Vision model QuickStart

    Features

    Generative AI

    Classic Models & Pipelines

    Deployment & Integration

    Hardware Acceleration

    → Full feature list

    Documentation

    TopicLink
    DeploymentDeploying the server
    Model repositoryPreparing models
    Client librariesWriting client code
    Demos & examplesDemos
    Release notesGitHub Releases

    Get the Server

    Docker images:

    docker pull openvino/model_server:latest        # Intel CPU
docker pull openvino/model_server:latest-gpu    # Intel CPU,GPU,NPU

docker pull openvino/model_server:weekly        # pre-production version with all accelerators enabled

    Binary official packages (Linux & Windows): GitHub Releases

    Binary pre-production packages (Linux & Windows): storage.openvinotoolkit.org

    Contributing

    Contributions are welcome! Please open an issue or pull request on GitHub.
    See security policy for responsible disclosure.

    References

    * Other names and brands may be claimed as the property of others.

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    ollama
    ollama·Go

    本地一键跑起 Kimi、DeepSeek、Qwen、Gemma 等大模型，省去环境配置和依赖折腾，适合快速做推理和原型验证。

    本地推理+3大模型推理服务入门
    186510+32019kv0.40.152026-11-10 01:49 UTC
    llama.cpp
    ggml-org·C++

    用 C/C++ 实现的大模型推理引擎，能在普通 CPU 和消费级显卡上跑 LLaMA 等模型，支持量化以降低显存占用。

    推理引擎+4大模型推理服务入门
    133898+30125kb122332026-11-10 10:59 UTC
    vllm
    vllm-project·Python

    面向大语言模型的高吞吐、低显存推理与服务引擎，通过 PagedAttention 等优化显著提升并发吞吐并降低显存占用，适合部署在线推理服务。

    推理引擎+4大模型推理服务进阶
    96670+17725kv0.33.0rc32026-11-10 11:13 UTC
    jan
    janhq·Rust

    在本地电脑上离线运行大模型的桌面客户端，是 ChatGPT 的开源替代品，无需联网即可对话，数据不出本机。

    本地推理+4大模型推理服务
    45651+1093.2kv0.8.82026-11-10 09:10 UTC