登录

JetStream

AI-HypercomputerApache-2.0更新放缓

面向 XLA 设备（先支持 TPU，后续扩展 GPU）的大模型推理引擎，主打高吞吐和低显存占用，适合在 JAX 生态里跑 Llama、Gemma 等模型。

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

v0.3发布于 2024-12-18 19:41 UTC473star70fork同步于 2026-11-07 07:31 UTC
推理引擎TPU 推理XLA高吞吐低显存
对比 GitHubgemmagptgpuinferencejax

怎么上手

由 README 提炼，执行前请确认
原文 
make install-deps
硬件要求按 README 说明

README 未说明具体显存要求；文档示例面向 Cloud TPU v5e，GPU 支持列为未来计划。

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

在 XLA 设备（先支持 TPU）上做 LLM 推理的吞吐与显存优化引擎

面向 XLA 设备（先支持 TPU，后续扩展 GPU）的大模型推理引擎，主打高吞吐和低显存占用，适合在 JAX 生态里跑 Llama、Gemma 等模型。

核心能力

  • 面向 XLA 设备优化 LLM 推理吞吐与内存
  • 提供 Jax（MaxText）参考引擎实现
  • 提供 PyTorch 参考引擎实现
  • 内置 mock 服务端与 requester、load_tester 压测工具
  • 支持 Prometheus 指标观测与 JAX Profiler 性能分析

容易踩的坑

项目将于 2026 年 2 月 1 日归档为只读，核心功能已迁移到 tpu-inference（tpu.vllm.ai）；Jax 与 PyTorch 引擎分别在独立仓库（maxtext、jetstream-pytorch）中，需另行获取；本地默认跑的是 mock 服务端，真实模型推理需按对应引擎文档配置。

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

4 个版本v0.3
全部 release
  1. v0.3最新正式版2024-12-18 · 2024-12-18 19:41 UTC

    Key Changes

    • Observability improvements in JetStream Server (prometheus metrics)
    • Tensorboard support for remote access
    • Engine API update for TTFT and TPOT measurements
    • Hugginface tokenizer support
    • Copybara G3 support
    • Threading optimizations

    What's Changed

  2. v0.2.22024-05-31 · 2024-05-31 18:44 UTC
    查看更新说明

    Key Changes

    • Enable observability in JetStream Server (prometheus metrics)
    • Enable JAX profiler support on single-host JetStream Server
    • Support both text and token ids I/O for JetStream Decode API
    • Add health check API
    • Support MLPerf evaluation
    • Enable JetStream Server E2E tests
    • Increase unit test coverage (>=96%)

    What's Changed

    New Contributors

    Full Changelog: https://github.com/google/JetStream/compare/v0.2.1...v0.2.2

  3. v0.2.12024-05-03 · 2024-05-03 21:23 UTC
    查看更新说明

    Key Changes

    • Support Llama3 tokenizer
    • JetStream Tokenizer refactor
    • Disaggregation preparation work

    What's Changed

    New Contributors

    Full Changelog: https://github.com/google/JetStream/compare/v0.2.0...v0.2.1

  4. v0.2.02024-04-05 · 2024-04-05 20:42 UTC
    查看更新说明

    Major Changes

    • Support JetStream MaxText inference on Cloud TPU VM
    • Support JetStream Pytorch inference on Cloud TPU VM
    • Support Continuous Batching with interleaved mode in JetStream
    • Support online serving benchmarking

    What's Changed

    New Contributors

    Full Changelog: https://github.com/google/JetStream/commits/v0.2.0

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    Unit Tests PyPI version PyPi downloads Contributions welcome

    [!WARNING] Notice of Archival: In an effort to streamline TPU inference efforts in open source, we have migrated core functionality in Jetstream to the new tpu-inference repository. For this reason, we will be archiving Jetstream on February 1st 2026. Please note, archival does not mean deletion! Users will still be able to fork and clone Jetstream, we are simply shifting the repository to "read-only". To get Jetstream features and so much more, please check out tpu.vllm.ai.

    JetStream is a throughput and memory optimized engine for LLM inference on XLA devices.

    About

    JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

    JetStream Engine Implementation

    Currently, there are two reference engine implementations available -- one for Jax models and another for Pytorch models.

    Jax

    Pytorch

    Documentation

    JetStream Standalone Local Setup

    Getting Started

    Setup

    make install-deps

    Run local server & Testing

    Use the following commands to run a server locally:

    # Start a server
python -m jetstream.core.implementations.mock.server

# Test local mock server
python -m jetstream.tools.requester

# Load test local mock server
python -m jetstream.tools.load_tester

    Test core modules

    # Test JetStream core orchestrator
python -m unittest -v jetstream.tests.core.test_orchestrator

# Test JetStream core server library
python -m unittest -v jetstream.tests.core.test_server

# Test JetStream lora adapter tensorstore
python -m unittest -v jetstream.tests.core.lora.test_adapter_tensorstore

# Test mock JetStream engine implementation
python -m unittest -v jetstream.tests.engine.test_mock_engine

# Test mock JetStream token utils
python -m unittest -v jetstream.tests.engine.test_token_utils
python -m unittest -v jetstream.tests.engine.test_utils

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    ollama
    ollama·Go

    本地一键跑起 Kimi、DeepSeek、Qwen、Gemma 等大模型，省去环境配置和依赖折腾，适合快速做推理和原型验证。

    本地推理+3大模型推理服务入门
    186510+32019kv0.40.152026-11-10 01:49 UTC
    llama.cpp
    ggml-org·C++

    用 C/C++ 实现的大模型推理引擎，能在普通 CPU 和消费级显卡上跑 LLaMA 等模型，支持量化以降低显存占用。

    推理引擎+4大模型推理服务入门
    133898+30125kb122332026-11-10 10:59 UTC
    vllm
    vllm-project·Python

    面向大语言模型的高吞吐、低显存推理与服务引擎，通过 PagedAttention 等优化显著提升并发吞吐并降低显存占用，适合部署在线推理服务。

    推理引擎+4大模型推理服务进阶
    96670+17725kv0.33.0rc32026-11-10 11:13 UTC
    jan
    janhq·Rust

    在本地电脑上离线运行大模型的桌面客户端，是 ChatGPT 的开源替代品，无需联网即可对话，数据不出本机。

    本地推理+4大模型推理服务
    45651+1093.2kv0.8.82026-11-10 09:10 UTC