awesome-llm-and-aigc

Awesome-llm-and-aigc

🚀🚀🚀 This repository lists some awesome public projects about Large Language Model(LLM), Vision Language Model(VLM), Vision Language Action(VLA), AI Generated Content(AIGC), the related Datasets and Applications.

Contents

• Awesome-llm-and-aigc
  • Summary
    • Frameworks
      • Official Version
        • Neural Network Architecture
        • Large Language Model
        • Large Vision Language Model
        • Vision Language Action
        • AI Generated Content
      • Performance Analysis and Visualization
      • Training and Fine-Tuning Framework
      • Reinforcement Learning Framework
      • LLM Inference Framework
        • LLM Inference and Serving Engine
        • High Performance Kernel Library
        • C and CPP Implementation
        • Triton Implementation
        • Python Implementation
        • Mojo Implementation
        • Rust Implementation
        • zig Implementation
        • Go Implementation
      • LLM Quantization Framework
      • Application Development Platform
      • RAG Framework
      • Vector Database
      • Memory Management
    • Awesome List
    • Paper Overview
    • Learning Resources
    • Community
  • Prompts
  • Open API
    • Python API
    • Rust API
    • Csharp API
    • Node.js API
  • Applications
    • IDE
    • Chatbot
    • Object Detection Field
    • Autonomous Driving Field
    • Robotics and Embodied AI
    • Code Assistant
    • Translator
    • Local knowledge Base
    • Long-Term Memory
    • Question Answering System
    • Academic Field
    • Medical Field
    • Mental Health Field
    • Legal Field
    • Financial Field
    • Math Field
    • Music Field
    • Speech and audio Field
    • Humor Generation
    • Animation Field
    • Food Field
    • PPT Field
    • Tool Learning
    • Adversarial Attack Field
    • Multi-Agent Collaboration
    • AI Avatar and Digital Human
    • GUI
  • Datasets
    • Awesome Datasets List
    • Open Datasets Platform
    • Humanoid Robotics Datasets
    • Text Datasets
    • Multimodal Datasets
    • SFT Datasets
    • Datasets Tools
      • Data Annotation
  • Blogs
  • Interview

Summary

• Frameworks

  • Official Version

    • Neural Network Architecture

      神经网络架构

      • Transformer : "Attention is All You Need". (arXiv 2017).

      • KAN : "KAN: Kolmogorov-Arnold Networks". (arXiv 2024).

      • FlashAttention : Fast and memory-efficient exact attention. "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness". (arXiv 2022).

    • Large Language Model

      大语言模型（LLM）

      • GPT-1 : "Improving Language Understanding by Generative Pre-Training". (cs.ubc.ca, 2018).

      • GPT-2 : "Language Models are Unsupervised Multitask Learners". (OpenAI blog, 2019). Better language models and their implications.

      • GPT-3 : "GPT-3: Language Models are Few-Shot Learners". (arXiv 2020).

      • InstructGPT : "Training language models to follow instructions with human feedback". (arXiv 2022). "Aligning language models to follow instructions". (OpenAI blog, 2022).

      • ChatGPT: Optimizing Language Models for Dialogue.

      • GPT-4: GPT-4 is OpenAI’s most advanced system, producing safer and more useful responses. "Sparks of Artificial General Intelligence: Early experiments with GPT-4". (arXiv 2023). "GPT-4 Architecture, Infrastructure, Training Dataset, Costs, Vision, MoE". (SemianAlysis, 2023).

      • Llama 2 : Inference code for LLaMA models. "LLaMA: Open and Efficient Foundation Language Models". (arXiv 2023). "Llama 2: Open Foundation and Fine-Tuned Chat Models". (ai.meta.com, 2023-07-18). (2023-07-18, Llama 2 is here - get it on Hugging Face).

      • Llama 3 : The official Meta Llama 3 GitHub site.

      • Qwen（通义千问） : The official repo of Qwen (通义千问) chat & pretrained large language model proposed by Alibaba Cloud.

      • Qwen3 : Qwen3 is the large language model series developed by Qwen team, Alibaba Cloud. "Qwen3: Think Deeper, Act Faster". (Qwen Blog). "Qwen2.5 Technical Report". (arXiv 2024). "Qwen2 Technical Report". (arXiv 2024).

      • DeepSeek-V3 : "DeepSeek-V3 Technical Report". (arXiv 2024).

      • DeepSeek-R1 : "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning". (arXiv 2025).

      • Open R1 : Fully open reproduction of DeepSeek-R1.

      • TinyZero : Clean, minimal, accessible reproduction of DeepSeek R1-Zero. TinyZero is a reproduction of DeepSeek R1 Zero in countdown and multiplication tasks. We built upon veRL.

      • GRPO-Zero : GRPO training with minimal dependencies. We implement almost everything from scratch and only depend on tokenizers for tokenization and pytorch for training.

      • Search-R1 : Search-R1: An Efficient, Scalable RL Training Framework for Reasoning & Search Engine Calling interleaved LLM based on veRL. "Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning". (arXiv 2025).

      • Logic-RL : Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning. "Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning". (arXiv 2025).

      • X-R1 : X-R1 aims to build an easy-to-use, low-cost training framework based on end-to-end reinforcement learning to accelerate the development of Scaling Post-Training. Inspired by DeepSeek-R1 and open-r1 , we produce minimal-cost for training 0.5B R1-Zero "Aha Moment"💡 from base model

      • DeepScaleR : Democratizing Reinforcement Learning for LLMs. www.agentica-project.com. "DeepScaleR: Surpassing O1-Preview with a 1.5B Model by Scaling RL"

      • OpenSeek : OpenSeek aims to unite the global open source community to drive collaborative innovation in algorithms, data and systems to develop next-generation models that surpass DeepSeek.

      • Gemma : The official PyTorch implementation of Google's Gemma models. ai.google.dev/gemma

      • Grok-1 : This repository contains JAX example code for loading and running the Grok-1 open-weights model.

      • Claude : Claude is a next-generation AI assistant based on Anthropic’s research into training helpful, honest, and harmless AI systems.

      • Whisper : Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification. "Robust Speech Recognition via Large-Scale Weak Supervision". (arXiv 2022).

      • OpenChat : OpenChat: Advancing Open-source Language Models with Imperfect Data. huggingface.co/openchat/openchat

      • GPT-Engineer : Specify what you want it to build, the AI asks for clarification, and then builds it. GPT Engineer is made to be easy to adapt, extend, and make your agent learn how you want your code to look. It generates an entire codebase based on a prompt.

      • StableLM : StableLM: Stability AI Language Models.

      • JARVIS : JARVIS, a system to connect LLMs with ML community. "HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace". (arXiv 2023).

      • MiniGPT-4 : MiniGPT-4: Enhancing Vision-language Understanding with Advanced Large Language Models. minigpt-4.github.io

      • minGPT : A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training.

      • nanoGPT : The simplest, fastest repository for training/finetuning medium-sized GPTs.

      • MicroGPT : A simple and effective autonomous agent compatible with GPT-3.5-Turbo and GPT-4. MicroGPT aims to be as compact and reliable as possible.

      • Dolly : Databricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform. Hello Dolly: Democratizing the magic of ChatGPT with open models

      • LMFlow : An extensible, convenient, and efficient toolbox for finetuning large machine learning models, designed to be user-friendly, speedy and reliable, and accessible to the entire community. Large Language Model for All. optimalscale.github.io/LMFlow/

      • Colossal-AI : Making big AI models cheaper, easier, and scalable. www.colossalai.org. "Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training". (arXiv 2021).

      • Lit-LLaMA : ⚡ Lit-LLaMA. Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. Apache 2.0-licensed.

      • GPT-4-LLM : "Instruction Tuning with GPT-4". (arXiv 2023). instruction-tuning-with-gpt-4.github.io/

      • Stanford Alpaca : Stanford Alpaca: An Instruction-following LLaMA Model.

      • Liger-Kernel : Efficient Triton Kernels for LLM Training. arxiv.org/pdf/2410.10989

      • FlagGems : FlagGems is a high-performance general operator library implemented in OpenAI Triton. It aims to provide a suite of kernel functions to accelerate LLM training and inference.

      • feizc/Visual-LLaMA : Open LLaMA Eyes to See the World. This project aims to optimize LLaMA model for visual information understanding like GPT-4 and further explore the potentional of large language model.

      • Lightning-AI/lightning-colossalai : Efficient Large-Scale Distributed Training with Colossal-AI and Lightning AI.

      • GPT4All : GPT4All: An ecosystem of open-source on-edge large language models. GTP4All is an ecosystem to train and deploy powerful and customized large language models that run locally on consumer grade CPUs.

      • ChatALL : Concurrently chat with ChatGPT, Bing Chat, bard, Alpaca, Vincuna, Claude, ChatGLM, MOSS, iFlytek Spark, ERNIE and more, discover the best answers. chatall.ai

      • 1595901624/gpt-aggregated-edition : 聚合ChatGPT官方版、ChatGPT免费版、文心一言、Poe、chatchat等多平台，支持自定义导入平台。

      • FreedomIntelligence/LLMZoo : ⚡LLM Zoo is a project that provides data, models, and evaluation benchmark for large language models.⚡ Tech Report

      • shm007g/LLaMA-Cult-and-More : News about 🦙 Cult and other AIGC models.

      • X-PLUG/mPLUG-Owl : mPLUG-Owl🦉: Modularization Empowers Large Language Models with Multimodality.

      • i-Code : The ambition of the i-Code project is to build integrative and composable multimodal Artificial Intelligence. The "i" stands for integrative multimodal learning. "CoDi: Any-to-Any Generation via Composable Diffusion". (arXiv 2023).

      • WorkGPT : WorkGPT is an agent framework in a similar fashion to AutoGPT or LangChain.

      • h2oGPT : h2oGPT is a large language model (LLM) fine-tuning framework and chatbot UI with document(s) question-answer capabilities. "h2oGPT: Democratizing Large Language Models". (arXiv 2023).

      • LongLLaMA : LongLLaMA is a large language model capable of handling long contexts. It is based on OpenLLaMA and fine-tuned with the Focused Transformer (FoT) method.

      • LLaMA-Adapter : Fine-tuning LLaMA to follow Instructions within 1 Hour and 1.2M Parameters. LLaMA-Adapter: Efficient Fine-tuning of LLaMA 🚀

      • DemoGPT : Create 🦜️🔗 LangChain apps by just using prompts with the power of Llama 2 🌟 Star to support our work! | 只需使用句子即可创建 LangChain 应用程序。 给个star支持我们的工作吧！DemoGPT: Auto Gen-AI App Generator with the Power of Llama 2. ⚡ With just a prompt, you can create interactive Streamlit apps via 🦜️🔗 LangChain's transformative capabilities & Llama 2.⚡ demogpt.io

      • Lamini : Lamini: The LLM engine for rapidly customizing models 🦙

      • xorbitsai/inference : Xorbits Inference (Xinference) is a powerful and versatile library designed to serve LLMs, speech recognition models, and multimodal models, even on your laptop. It supports a variety of models compatible with GGML, such as llama, chatglm, baichuan, whisper, vicuna, orac, and many others.

      • epfLLM/Megatron-LLM : distributed trainer for LLMs.

      • AmineDiro/cria : OpenAI compatible API for serving LLAMA-2 model.

      • Llama-2-Onnx : Llama 2 Powered By ONNX.

      • gpt-llm-trainer : The goal of this project is to explore an experimental new pipeline to train a high-performing task-specific model. We try to abstract away all the complexity, so it's as easy as possible to go from idea -> performant fully-trained model.

      • ChatGLM-6B : ChatGLM-6B: An Open Bilingual Dialogue Language Model | 开源双语对话语言模型。 ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 General Language Model (GLM) 架构，具有 62 亿参数。 "GLM: General Language Model Pretraining with Autoregressive Blank Infilling". (ACL 2022). "GLM-130B: An Open Bilingual Pre-trained Model". (ICLR 2023).

      • ChatGLM2-6B : ChatGLM2-6B: An Open Bilingual Chat LLM | 开源双语对话语言模型。ChatGLM2-6B 是开源中英双语对话模型 ChatGLM-6B 的第二代版本，在保留了初代模型对话流畅、部署门槛较低等众多优秀特性的基础之上，ChatGLM2-6B 引入了更强大的性能、更强大的性能、更高效的推理、更开放的协议。

      • ChatGLM3 : ChatGLM3 series: Open Bilingual Chat LLMs | 开源双语对话语言模型。

      • InternLM（书生·浦语） : Official release of InternLM2 7B and 20B base and chat models. 200K context support. internlm.intern-ai.org.cn/

      • Baichuan-7B（百川-7B） : A large-scale 7B pretraining language model developed by BaiChuan-Inc. Baichuan-7B 是由百川智能开发的一个开源可商用的大规模预训练语言模型。基于 Transformer 结构，在大约 1.2 万亿 tokens 上训练的 70 亿参数模型，支持中英双语，上下文窗口长度为 4096。在标准的中文和英文 benchmark（C-Eval/MMLU）上均取得同尺寸最好的效果。huggingface.co/baichuan-inc/baichuan-7B

      • Baichuan-13B（百川-13B） : A 13B large language model developed by Baichuan Intelligent Technology. Baichuan-13B 是由百川智能继 Baichuan-7B 之后开发的包含 130 亿参数的开源可商用的大规模语言模型，在权威的中文和英文 benchmark 上均取得同尺寸最好的效果。本次发布包含有预训练 (Baichuan-13B-Base) 和对齐 (Baichuan-13B-Chat) 两个版本。huggingface.co/baichuan-inc/Baichuan-13B-Chat

      • Baichuan2 : A series of large language models developed by Baichuan Intelligent Technology. Baichuan 2 是百川智能推出的新一代开源大语言模型，采用 2.6 万亿 Tokens 的高质量语料训练。Baichuan 2 在多个权威的中文、英文和多语言的通用、领域 benchmark 上取得同尺寸最佳的效果。本次发布包含有 7B、13B 的 Base 和 Chat 版本，并提供了 Chat 版本的 4bits 量化。huggingface.co/baichuan-inc. "Baichuan 2: Open Large-scale Language Models". (arXiv 2023).

      • MOSS : An open-source tool-augmented conversational language model from Fudan University. MOSS是一个支持中英双语和多种插件的开源对话语言模型，moss-moon系列模型具有160亿参数，在FP16精度下可在单张A100/A800或两张3090显卡运行，在INT4/8精度下可在单张3090显卡运行。MOSS基座语言模型在约七千亿中英文以及代码单词上预训练得到，后续经过对话指令微调、插件增强学习和人类偏好训练具备多轮对话能力及使用多种插件的能力。txsun1997.github.io/blogs/moss.html

      • BayLing（百聆） : “百聆”是一个具有增强的语言对齐的英语/中文大语言模型，具有优越的英语/中文能力，在多项测试中取得ChatGPT 90%的性能。BayLing is an English/Chinese LLM equipped with advanced language alignment, showing superior capability in English/Chinese generation, instruction following and multi-turn interaction. nlp.ict.ac.cn/bayling. "BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models". (arXiv 2023).

      • FlagAI（悟道·天鹰（Aquila）） : FlagAI (Fast LArge-scale General AI models) is a fast, easy-to-use and extensible toolkit for large-scale model. Our goal is to support training, fine-tuning, and deployment of large-scale models on various downstream tasks with multi-modality.

      • YuLan-Chat（玉兰） : YuLan-Chat models are chat-based large language models, which are developed by the researchers in GSAI, Renmin University of China (YuLan, which represents Yulan Magnolia, is the campus flower of Renmin University of China). The newest version is developed by continually-pretraining and instruction-tuning LLaMA-2 with high-quality English and Chinese data. YuLan-Chat系列模型是中国人民大学高瓴人工智能学院师生共同开发的支持聊天的大语言模型（名字"玉兰"取自中国人民大学校花）。 最新版本基于LLaMA-2进行了中英文双语的继续预训练和指令微调。

      • Yi-1.5 : Yi-1.5 is an upgraded version of Yi, delivering stronger performance in coding, math, reasoning, and instruction-following capability.

      • 智海-录问 : 智海-录问(wisdomInterrogatory)是由浙江大学、阿里巴巴达摩院以及华院计算三家单位共同设计研发的法律大模型。核心思想：以“普法共享和司法效能提升”为目标，从推动法律智能化体系入司法实践、数字化案例建设、虚拟法律咨询服务赋能等方面提供支持，形成数字化和智能化的司法基座能力。

      • 活字 : 活字是由哈工大自然语言处理研究所多位老师和学生参与开发的一个开源可商用的大规模预训练语言模型。 该模型基于 Bloom 结构的70 亿参数模型，支持中英双语，上下文窗口长度为 2048。 在标准的中文和英文基准以及主观评测上均取得同尺寸中优异的结果。

      • MiLM-6B : MiLM-6B 是由小米开发的一个大规模预训练语言模型，参数规模为64亿。在 C-Eval 和 CMMLU 上均取得同尺寸最好的效果。

      • Chinese LLaMA and Alpaca : 中文LLaMA&Alpaca大语言模型+本地CPU/GPU训练部署 (Chinese LLaMA & Alpaca LLMs)。"Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca". (arXiv 2023).

      • Chinese-LLaMA-Alpaca-2 : 中文 LLaMA-2 & Alpaca-2 大模型二期项目 (Chinese LLaMA-2 & Alpaca-2 LLMs).

      • FlagAlpha/Llama2-Chinese : Llama中文社区，最好的中文Llama大模型，完全开源可商用。

      • michael-wzhu/Chinese-LlaMA2 : Repo for adapting Meta LlaMA2 in Chinese! META最新发布的LlaMA2的汉化版！ （完全开源可商用）

      • CPM-Bee : CPM-Bee是一个完全开源、允许商用的百亿参数中英文基座模型，也是CPM-Live训练的第二个里程碑。

      • PandaLM : PandaLM: Reproducible and Automated Language Model Assessment.

      • SpeechGPT : "SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities". (arXiv 2023).

      • GPT2-Chinese : Chinese version of GPT2 training code, using BERT tokenizer.

      • Chinese-Tiny-LLM : "Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model". (arXiv 2024).

      • 潘多拉 (Pandora) : 潘多拉，一个让你呼吸顺畅的ChatGPT。Pandora, a ChatGPT that helps you breathe smoothly.

      • 百度-文心大模型 : 百度全新一代知识增强大语言模型，文心大模型家族的新成员，能够与人对话互动，回答问题，协助创作，高效便捷地帮助人们获取信息、知识和灵感。

      • 百度智能云-千帆大模型 : 百度智能云千帆大模型平台一站式企业级大模型平台，提供先进的生成式AI生产及应用全流程开发工具链。

      • 华为云-盘古大模型 : 盘古大模型致力于深耕行业，打造金融、政务、制造、矿山、气象、铁路等领域行业大模型和能力集，将行业知识know-how与大模型能力相结合，重塑千行百业，成为各组织、企业、个人的专家助手。"Accurate medium-range global weather forecasting with 3D neural networks". (Nature 2023).

      • 商汤科技-日日新SenseNova : 日日新（SenseNova），是商汤科技宣布推出的大模型体系，包括自然语言处理模型“商量”（SenseChat）、文生图模型“秒画”和数字人视频生成平台“如影”（SenseAvatar）等。

      • 科大讯飞-星火认知大模型 : 新一代认知智能大模型，拥有跨领域知识和语言理解能力，能够基于自然对话方式理解与执行任务。

      • 字节跳动-豆包 : 豆包。

      • CrazyBoyM/llama3-Chinese-chat : Llama3 中文版。

    • Large Vision Language Model

      视觉语言大模型（LVLM）

      • Qwen2.5-VL : Qwen2-VL is the multimodal large language model series developed by Qwen team, Alibaba Cloud. "Qwen2.5-VL Technical Report". (arXiv 2025). 2025-01-26，Qwen2.5 VL! Qwen2.5 VL! Qwen2.5 VL!. "Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution". (arXiv 2024). "Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond". (arXiv 2023).

      • Kimi-VL : Kimi-VL: Mixture-of-Experts Vision-Language Model for Multimodal Reasoning, Long-Context Understanding, and Strong Agent Capabilities. "Kimi-VL Technical Report". (arXiv 2025).

      • Visual-RFT : 🌈We introduce Visual Reinforcement Fine-tuning (Visual-RFT), the first comprehensive adaptation of Deepseek-R1's RL strategy to the multimodal field. We use the Qwen2-VL-2/7B model as our base model and design a rule-based verifiable reward, which is integrated into a GRPO-based reinforcement fine-tuning framework to enhance the performance of LVLMs across various visual perception tasks. ViRFT extends R1's reasoning capabilities to multiple visual perception tasks, including various detection tasks like Open Vocabulary Detection, Few-shot Detection, Reasoning Grounding, and Fine-grained Image Classification. "Visual-RFT: Visual Reinforcement Fine-Tuning". (arXiv 2025).

      • VLM-R1 : VLM-R1: A stable and generalizable R1-style Large Vision-Language Model. Solve Visual Understanding with Reinforced VLMs. 2025-03-20，Improving Object Detection through Reinforcement Learning with VLM-R1.

      • Video-R1 : "Video-R1: Reinforcing Video Reasoning in MLLMs". (arXiv 2025).

      • MAYE : This project presents MAYE, a transparent and reproducible framework and a comprehensive evaluation scheme for applying reinforcement learning (RL) to vision-language models (VLMs). The codebase is built entirely from scratch without relying on existing RL toolkits. "Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme". (arXiv 2025).

      • Osilly/Vision-R1 : "Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models". (arXiv 2025).

      • Griffon/Vision-R1 : "Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning". (arXiv 2025).

      • Janus : 🚀 Janus-Series: Unified Multimodal Understanding and Generation Models. "Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling". (arXiv 2025). "Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation". (arXiv 2024). "JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation". (arXiv 2024).

      • VisualThinker-R1-Zero : VisualThinker-R1-Zero: First ever R1-Zero's Aha Moment on just a 2B non-SFT Model. VisualThinker-R1-Zero is a replication of DeepSeek-R1-Zero in visual reasoning. We are the first to successfully observe the emergent “aha moment” and increased response length in visual reasoning on just a 2B non-SFT models. For more details, please refer to the notion report.

      • R1-V : R1-V: Reinforcing Super Generalization Ability in Vision Language Models with Less Than $3.

      • LLaVA : 🌋 LLaVA: Large Language and Vision Assistant. Visual instruction tuning towards large language and vision models with GPT-4 level capabilities. llava.hliu.cc. "Visual Instruction Tuning". (arXiv 2023).

      • NVILA : VILA - a multi-image visual language model with training, inference and evaluation recipe, deployable from cloud to edge (Jetson Orin and laptops). "NVILA: Efficient Frontier Visual Language Models". (arXiv 2024).

      • Visual ChatGPT : Visual ChatGPT connects ChatGPT and a series of Visual Foundation Models to enable sending and receiving images during chatting. "Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models". (arXiv 2023).

      • CLIP : CLIP (Contrastive Language-Image Pretraining), Predict the most relevant text snippet given an image. "Learning Transferable Visual Models From Natural Language Supervision". (arXiv 2021).

      • OpenCLIP : Welcome to an open source implementation of OpenAI's CLIP (Contrastive Language-Image Pre-training). "Reproducible scaling laws for contrastive language-image learning". (arXiv 2022).

      • GLIP : "Grounded Language-Image Pre-training". (CVPR 2022).

      • GLIPv2 : "GLIPv2: Unifying Localization and Vision-Language Understanding". (arXiv 2022).

      • InternImage : "InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions". (CVPR 2023).

      • SAM : The repository provides code for running inference with the Segment Anything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model. "Segment Anything". (arXiv 2023).

      • Grounded-SAM : Marrying Grounding DINO with Segment Anything & Stable Diffusion & Tag2Text & BLIP & Whisper & ChatBot - Automatically Detect , Segment and Generate Anything with Image, Text, and Audio Inputs. We plan to create a very interesting demo by combining Grounding DINO and Segment Anything which aims to detect and segment Anything with text inputs!

      • SEEM : We introduce SEEM that can Segment Everything Everywhere with Multi-modal prompts all at once. SEEM allows users to easily segment an image using prompts of different types including visual prompts (points, marks, boxes, scribbles and image segments) and language prompts (text and audio), etc. It can also work with any combinations of prompts or generalize to custom prompts! "Segment Everything Everywhere All at Once". (arXiv 2023).

      • SAM3D : "SAM3D: Zero-Shot 3D Object Detection via Segment Anything Model". (arXiv 2023).

      • ImageBind : "ImageBind: One Embedding Space To Bind Them All". (CVPR 2023).

      • Track-Anything : Track-Anything is a flexible and interactive tool for video object tracking and segmentation, based on Segment Anything, XMem, and E2FGVI. "Track Anything: Segment Anything Meets Videos". (arXiv 2023).

      • qianqianwang68/omnimotion : "Tracking Everything Everywhere All at Once". (arXiv 2023).

      • M3I-Pretraining : "Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Information". (arXiv 2022).

      • BEVFormer : BEVFormer: a Cutting-edge Baseline for Camera-based Detection. "BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers". (arXiv 2022).

      • Uni-Perceiver : "Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks". (CVPR 2022).

      • AnyLabeling : 🌟 AnyLabeling 🌟. Effortless data labeling with AI support from YOLO and Segment Anything! Effortless data labeling with AI support from YOLO and Segment Anything!

      • X-AnyLabeling : 💫 X-AnyLabeling 💫. Effortless data labeling with AI support from Segment Anything and other awesome models!

      • Label Anything : OpenMMLab PlayGround: Semi-Automated Annotation with Label-Studio and SAM.

      • RevCol : "Reversible Column Networks". (arXiv 2023).

      • Macaw-LLM : Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration.

      • SAM-PT : SAM-PT: Extending SAM to zero-shot video segmentation with point-based tracking. "Segment Anything Meets Point Tracking". (arXiv 2023).

      • Video-LLaMA : "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding". (arXiv 2023).

      • Video-LLaVA : "Video-LLaVA: Learning United Visual Representation by Alignment Before Projection". (EMNLP 2024).

      • MobileSAM : "Faster Segment Anything: Towards Lightweight SAM for Mobile Applications". (arXiv 2023).

      • BuboGPT : "BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs". (arXiv 2023).

    • Vision Language Action

      视觉语言动作大模型（VLA）

      • Embodied-R : "Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learning". (arXiv 2025).

    • AI Generated Content

      人工智能生成内容（AIGC）

      • Wan2.1 : Wan: Open and Advanced Large-Scale Video Generative Models.

      • Sora : Sora is an AI model that can create realistic and imaginative scenes from text instructions.

      • Open Sora Plan : This project aim to reproducing Sora (Open AI T2V model), but we only have limited resource. We deeply wish the all open source community can contribute to this project. 本项目希望通过开源社区的力量复现Sora，由北大-兔展AIGC联合实验室共同发起，当前我们资源有限仅搭建了基础架构，无法进行完整训练，希望通过开源社区逐步增加模块并筹集资源进行训练，当前版本离目标差距巨大，仍需持续完善和快速迭代，欢迎Pull request！！！Project Page 中文主页

      • Mini Sora : The Mini Sora project aims to explore the implementation path and future development direction of Sora.

      • EMO : "EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions". (arXiv 2024).

      • Stable Diffusion : Stable Diffusion is a latent text-to-image diffusion model. Stable Diffusion was made possible thanks to a collaboration with Stability AI and Runway and builds upon our previous work "High-Resolution Image Synthesis with Latent Diffusion Models". (CVPR 2022).

      • Stable Diffusion Version 2 : This repository contains Stable Diffusion models trained from scratch and will be continuously updated with new checkpoints. "High-Resolution Image Synthesis with Latent Diffusion Models". (CVPR 2022).

      • StableStudio : StableStudio by Stability AI. 👋 Welcome to the community repository for StableStudio, the open-source version of DreamStudio.

      • AudioCraft : Audiocraft is a library for audio processing and generation with deep learning. It features the state-of-the-art EnCodec audio compressor / tokenizer, along with MusicGen, a simple and controllable music generation LM with textual and melodic conditioning.

      • InvokeAI : Invoke AI - Generative AI for Professional Creatives. Professional Creative Tools for Stable Diffusion, Custom-Trained Models, and more. invoke-ai.github.io/InvokeAI/

      • DragGAN : "Stable Diffusion Training with MosaicML. This repo contains code used to train your own Stable Diffusion model on your own data". (SIGGRAPH 2023).

      • AudioGPT : AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head.

      • PandasAI : Pandas AI is a Python library that adds generative artificial intelligence capabilities to Pandas, the popular data analysis and manipulation tool. It is designed to be used in conjunction with Pandas, and is not a replacement for it.

      • mosaicml/diffusion : Stable Diffusion Training with MosaicML. This repo contains code used to train your own Stable Diffusion model on your own data.

      • VisorGPT : Customize spatial layouts for conditional image synthesis models, e.g., ControlNet, using GPT. "VisorGPT: Learning Visual Prior via Generative Pre-Training". (arXiv 2023).

      • ControlNet : Let us control diffusion models! "Adding Conditional Control to Text-to-Image Diffusion Models". (arXiv 2023).

      • Fooocus : Fooocus is an image generating software. Fooocus is a rethinking of Stable Diffusion and Midjourney’s designs. "微信公众号「GitHubStore」《Fooocus : 集Stable Diffusion 和 Midjourney 优点于一身的开源AI绘图软件》"。

      • MindDiffuser : "MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion". (arXiv 2023).

      • World Labs : We are a spatial intelligence company building Large World Models to perceive, generate, and interact with the 3D world.

      • Genie 2 : Genie 2: A large-scale foundation world model.

      • Midjourney : Midjourney is an independent research lab exploring new mediums of thought and expanding the imaginative powers of the human species.

      • DreamStudio : Effortless image generation for creators with big dreams.

      • Firefly : Adobe Firefly: Experiment, imagine, and make an infinite range of creations with Firefly, a family of creative generative AI models coming to Adobe products.

      • Jasper : Meet Jasper. On-brand AI content wherever you create.

      • Copy.ai : Whatever you want to ask, our chat has the answers.

      • Peppertype.ai : Leverage the AI-powered platform to ideate, create, distribute, and measure your content and prove your content marketing ROI.

      • ChatPPT : ChatPPT来袭命令式一键生成PPT。

  • Performance Analysis and Visualization

    性能分析及可视化

    • FlagPerf : FlagPerf is an open-source software platform for benchmarking AI chips. FlagPerf是智源研究院联合AI硬件厂商共建的一体化AI硬件评测引擎，旨在建立以产业实践为导向的指标体系，评测AI硬件在软件栈组合（模型+框架+编译器）下的实际能力。

    • hahnyuan/LLM-Viewer : Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline model in a user-friendly interface.

    • harleyszhang/llm_counts : llm theoretical performance analysis tools and support params, flops, memory and latency analysis.

  • Training and Fine-Tuning Framework

    训练和微调框架

    • DeepSpeed : DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective. www.deepspeed.ai/

    • unsloth : Finetune Llama 3.3, DeepSeek-R1 & Reasoning LLMs 2x faster with 70% less memory. unsloth.ai

    • LLaMA-Factory : Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024). "LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models". (arXiv 2024).

  • Reinforcement Learning Framework

    强化学习框架

    • TTRL : "TTRL: Test-Time Reinforcement Learning". (arXiv 2025).

  • LLM Inference Framework

    大语言模型推理框架

    • LLM Inference and Serving Engin