登录

GenerativeAIExamples

NVIDIAApache-2.0活跃维护

提供基于 GPU 加速和微服务架构的生成式 AI 参考工作流，涵盖 RAG、LLM 推理等场景，帮你在 Triton、TensorRT 等基础设施上快速搭建可落地的服务。

Generative AI reference workflows optimized for accelerated infrastructure and microservice architecture.

v0.8.0发布于 2024-08-21 03:11 UTC4,226star +2 / 7 天1,101fork同步于 2026-11-10 05:38 UTC
生成式AIRAGLLM推理微服务GPU加速
对比 GitHubgpu-accelerationlarge-language-modelsllmllm-inferencemicroservice

怎么上手

由 README 提炼，执行前请确认
原文 
export NVIDIA_API_KEY=nvapi-...
git clone https://github.com/nvidia/GenerativeAIExamples.git
cd GenerativeAIExamples/RAG/examples/basic_rag/langchain/
docker compose up -d --build
硬件要求按 README 说明

README 未说明具体显存要求；示例默认调用 NVIDIA API Catalog 上的 NIM 端点，也可按文档改为本地部署

有 Docker 镜像
下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

NVIDIA 官方生成式 AI 参考工作流集合，帮你快速搭建 RAG、Agent 和微调系统

提供基于 GPU 加速和微服务架构的生成式 AI 参考工作流，涵盖 RAG、LLM 推理等场景，帮你在 Triton、TensorRT 等基础设施上快速搭建可落地的服务。

核心能力

  • 提供基于 LangChain 和 LlamaIndex 的端到端 RAG 示例
  • 用 docker compose 一键启动基础 RAG 流水线与 Playground
  • 覆盖多轮对话、多模态、结构化数据、查询分解等进阶 RAG
  • 集成 NeMo 微服务做工具调用微调、评估与护栏
  • 提供视觉 NIM 工作流，含 VLM 告警与 NV-CLIP 图文检索

容易踩的坑

需先到 NVIDIA API Catalog 申请 API key 并导出环境变量，否则示例无法调用模型；视觉工作流需用 --recurse-submodules 递归克隆，否则子模块目录为空；默认通过 https://localhost:8090/ 访问 Playground，端口被占用时需调整

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本v0.8.0
全部 release
  1. v0.8.0最新正式版2024-08-21 · 2024-08-21 03:11 UTC

    This release completely refactors the directory structure of the repository for a more seamless and intuitive developer journey. It also adds support to deploy the latest accelerated embedding and reranking models across the cloud, data center, and workstation using NVIDIA NeMo Retriever NIM microservices.

    Added

    Changed

    • Major restructuring and reorganisation of the assets within the repository
      • Top level experimental directory has been renamed as community.
      • Top level RetrievalAugmentedGeneration directory has been renamed as just RAG.
      • The Docker Compose files inside top level deploy directory has been migrated to example-specific directories under RAG/examples. The vector database and on-prem NIM microservices deployment files are under RAG/examples/local_deploy.
      • Top level models has been renamed to finetuning.
      • Top level notebooks directory has been moved to under RAG/notebooks and has been organised framework wise.
      • Top level tools directory has been migrated to RAG/tools.
      • Top level integrations directory has been moved into RAG/src.
      • RetreivalAugmentedGeneration/common is now residing under RAG/src/chain_server.
      • RetreivalAugmentedGeneration/frontend is now residing under RAG/src/rag_playground/default.
  2. v0.7.02024-06-18 · 2024-06-18 15:52 UTC
    查看更新说明

    This release switches all examples to use cloud hosted GPU accelerated LLM and embedding models from Nvidia API Catalog as default. It also deprecates support to deploy on-prem models using NeMo Inference Framework Container and adds support to deploy accelerated generative AI models across the cloud, data center, and workstation using latest Nvidia NIM-LLM.

    Added

    Changed

    • All examples now use llama3 models from Nvidia API Catalog as default. Summary of updated examples and the model it uses is available here.
    • Switched default embedding model of all examples to Snowflake arctic-embed-I model
    • Added more verbose logs and support to configure log level for chain server using LOG_LEVEL enviroment variable.
    • Bumped up version of langchain-nvidia-ai-endpoints, sentence-transformers package and milvus containers
    • Updated base containers to use ubuntu 22.04 image nvcr.io/nvidia/base/ubuntu:22.04_20240212
    • Added llama-index-readers-file as dependency to avoid runtime package installation within chain server.

    Deprecated

  3. v0.6.02024-05-10 · 2024-05-10 17:19 UTC
    查看更新说明

    This release adds ability to switch between API Catalog models and on-prem models using NIM-LLM and adds documentation on how to build an RAG application from scratch. It also releases a containerized end to end RAG evaluation application integrated with RAG chain-server APIs.

    Added

    Changed

    • Renamed example csv_rag to structured_data_rag
    • Model Engine name update
      • nv-ai-foundation and nv-api-catalog llm engine are renamed to nvidia-ai-endpoints
      • nv-ai-foundation embedding engine is renamed to nvidia-ai-endpoints
    • Embedding model update
      • developer_rag example uses UAE-Large-V1 embedding model.
      • Using ai-embed-qa-4 for api catalog examples instead of nvolveqa_40k as embedding model
    • Ingested data now persists across multiple sessions.
    • Updated langchain-nvidia-endpoints to version 0.0.11, enabling support for models like llama3.
    • File extension based validation to throw error for unsupported files.
    • The default output token length in the UI has been increased from 250 to 1024 for more comprehensive responses.
    • Stricter chain-server API validation support to enhance API security
    • Updated version of llama-index, pymilvus.
    • Updated pgvector container to pgvector/pgvector:pg16
    • LLM Model Updates
  4. v0.5.02024-03-20 · 2024-03-20 18:10 UTC
    查看更新说明

    This release adds new dedicated RAG examples showcasing state of the art usecases, switches to the latest API catalog endpoints from NVIDIA and also refactors the API interface of chain-server. This release also improves the developer experience by adding github pages based documentation and streamlining the example deployment flow using dedicated compose files.

    Added

    Changed

    • Switched from NVIDIA AI Foundation to NVIDIA API Catalog endpoints for accessing cloud hosted LLM models.
    • Refactored API schema of chain-server component to support runtime allocation of llm parameters like temperature, max tokens, chat history etc.
    • Renamed llm-playground service in compose files to rag-playground.
    • Switched base containers for all components to ubuntu instead of pytorch and optimized container build time as well as container size.
    • Deprecated yaml based configuration to avoid confusion, all configurations are now environment variable based.
    • Removed requirement of hardcoding NVIDIA_API_KEY in compose.env file.
    • Upgraded all python dependencies for chain-server and rag-playground services.

    Fixed

    • Fixed a bug causing hallucinated answer when retriever fails to return any documents.
    • Fixed some accuracy issues for all the examples.
  5. v0.4.02024-02-22 · 2024-02-22 20:51 UTC
    查看更新说明

    This release adds new dedicated notebooks showcasing usage of cloud based NVIDIA AI Foundation models, upgraded milvus container version to enable GPU accelerated vector search and added support for FAISS vector database. Detailed changes are listed below:

    Added

    • New dedicated notebooks showcasing usage of cloud based Nvidia AI Foundation based models using Langchain connectors as well as local model deployment using Huggingface.
    • Upgraded milvus container version to enable GPU accelerated vector search.
    • Added support to interact with models behind NeMo Inference Microservices using new model engines nemo-embed and nemo-infer.
    • Added support to provide example specific collection name for vector databases using an environment variable named COLLECTION_NAME.
    • Added faiss as a generic vector database solution behind utils.py.

    Changed

    • Upgraded and changed base containers for all components to pytorch 23.12-py3.
    • Added langchain specific vector database connector in utils.py.
    • Changed speech support to use single channel for Riva ASR and TTS.
    • Changed get_llm utility in utils.py to return Langchain wrapper instead of Llmaindex wrappers.

    Fixed

    • Fixed a bug causing empty rating in evaluation notebook
    • Fixed document search implementation of query decomposition example.

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    NVIDIA Generative AI Examples

    This repository is a starting point for developers looking to integrate with the NVIDIA software ecosystem to speed up their generative AI systems. Whether you are building RAG pipelines, agentic workflows, or fine-tuning models, this repository will help you integrate NVIDIA, seamlessly and natively, with your development stack.

    Table of Contents

    What's New?

    Data Flywheel

    These tutorials demonstrate Data Flywheel workflows that use NVIDIA NeMo Microservices. They include components such as NVIDIA NeMo Datastore, NeMo Entity Store, NeMo Customizer, NeMo Evaluator, NeMo Guardrails microservices, and NVIDIA NIMs.

    Safer Agentic AI

    The following tutorials illustrate how to audit your large language models with NeMo Auditor to identify vulnerabilities to unsafe prompts, and how to run inference with multiple rails in parallel to reduce latency and improve throughput.

    Knowledge Graph RAG

    This example implements a GPU-accelerated pipeline for creating and querying knowledge graphs using RAG by leveraging NIM microservices and the RAPIDS ecosystem to process large-scale datasets efficiently.

    Agentic Workflows with Llama 3.1

    RAG with Local NIM Deployment and LangChain

    • Tips for Building a RAG Pipeline with NVIDIA AI LangChain AI Endpoints by Amit Bleiweiss. [Blog, Notebook]

    For more information, refer to the Generative AI Example releases.

    Vision NIM Workflows

    A collection of Jupyter notebooks, sample code and reference applications built with Vision NIMs.

    To pull the vision NIM workflows, clone this repository recursively:

    git clone https://github.com/nvidia/GenerativeAIExamples --recurse-submodules

    The workflows will then be located at GenerativeAIExamples/vision_workflows

    Follow the links below to learn more:

    Try it Now!

    Experience NVIDIA RAG Pipelines with just a few steps!

    1. Get your NVIDIA API key.

      1. Go to the NVIDIA API Catalog.
      2. Select any model.
      3. Click Get API Key.
      4. Run: 
        export NVIDIA_API_KEY=nvapi-...

    2. Clone the repository.

      git clone https://github.com/nvidia/GenerativeAIExamples.git

    3. Build and run the basic RAG pipeline.

      cd GenerativeAIExamples/RAG/examples/basic_rag/langchain/
docker compose up -d --build

    4. Go to https://localhost:8090/ and submit queries to the sample RAG Playground.

    5. Stop containers when done.

      docker compose down

    Data Flywheel

    A Data Flywheel is a self-reinforcing cycle where user interactions generate data that improves AI models or products, leading to better outcomes that attract more users and further enhance data quality. This feedback loop relies on continuous data processing, model refinement, and guardrails to ensure accuracy and compliance while compounding value over time. Real-world applications range from personalized customer experiences to operational systems like inventory management, where improved predictions drive efficiency and growth.

    Tool-Calling Notebooks

    Tool calling empowers Large Language Models (LLMs) to integrate with external APIs, execute dynamic workflows, and retrieve real-time data beyond their training scope. The NVIDIA NeMo microservices platform offers a modular infrastructure for deploying AI pipelines that includes fine-tuning, evaluation, inference, and guardrail enforcement—across Kubernetes clusters in cloud or on-premises environments.

    This end-to-end tutorial demonstrates how to leverage NeMo Microservices to customize Llama-3.2-1B-Instruct by using the xLAM function-calling dataset, assess its accuracy, and implement safety constraints to govern its behavior.

    RAG

    RAG Notebooks

    NVIDIA has first-class support for popular generative AI developer frameworks like LangChain, LlamaIndex, and Haystack. These end-to-end notebooks show how to integrate NIM microservices using your preferred generative AI development framework.

    Use these notebooks to learn about the LangChain and LlamaIndex connectors.

    LangChain Notebooks
    LlamaIndex Notebooks

    RAG Examples

    By default, these end-to-end examples use preview NIM endpoints on NVIDIA API Catalog. Alternatively, you can run any of the examples on premises.

    Basic RAG Examples
    Advanced RAG Examples

    RAG Tools

    Example tools and tutorials to enhance LLM development and productivity when using NVIDIA RAG pipelines.

    RAG Projects

    Documentation

    Getting Started

    How To's

    Reference

    Community

    We're posting these examples on GitHub to support the NVIDIA LLM community and facilitate feedback. We invite contributions! Open a GitHub issue or pull request! See contributing Check out the community examples and notebooks.

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    ollama
    ollama·Go

    本地一键跑起 Kimi、DeepSeek、Qwen、Gemma 等大模型，省去环境配置和依赖折腾，适合快速做推理和原型验证。

    本地推理+3大模型推理服务入门
    186472+28219kv0.40.152026-11-10 01:49 UTC
    llama.cpp
    ggml-org·C++

    用 C/C++ 实现的大模型推理引擎，能在普通 CPU 和消费级显卡上跑 LLaMA 等模型，支持量化以降低显存占用。

    推理引擎+4大模型推理服务入门
    133865+26825kb122332026-11-10 01:44 UTC
    vllm
    vllm-project·Python

    面向大语言模型的高吞吐、低显存推理与服务引擎，通过 PagedAttention 等优化显著提升并发吞吐并降低显存占用，适合部署在线推理服务。

    推理引擎+4大模型推理服务进阶
    96643+15025kv0.33.0rc32026-11-10 05:17 UTC
    jan
    janhq·Rust

    在本地电脑上离线运行大模型的桌面客户端，是 ChatGPT 的开源替代品，无需联网即可对话，数据不出本机。

    本地推理+4大模型推理服务
    45644+1023.2kv0.8.82026-11-10 04:20 UTC