Awesome-LLMOps

Awesome LLMOps

An awesome & curated list of the best LLMOps tools for developers.

[!NOTE] Contributions are most welcome, please adhere to the contribution guidelines.

Table of Contents

• Awesome LLMOps
  • Table of Contents
  • Model
    • Large Language Model
    • CV Foundation Model
    • Audio Foundation Model
    • Robotics Foundation Model
  • Serving
    • Large Model Serving
    • Frameworks/Servers for Serving
  • Security
    • Frameworks for LLM security
    • Observability
  • LLMOps
  • Search
    • Vector search
  • Code AI
  • Training
    • IDEs and Workspaces
    • Foundation Model Fine Tuning
    • Frameworks for Training
    • Experiment Tracking
    • Visualization
    • Model Editing
  • Data
    • Data Management
    • Data Storage
    • Data Tracking
    • Feature Engineering
    • Data/Feature enrichment
  • Large Scale Deployment
    • ML Platforms
    • Workflow
    • Scheduling
    • Model Management
  • Performance
    • ML Compiler
    • Profiling
  • AutoML
  • Optimizations
  • Federated ML
  • Awesome Lists

Model

Large Language Model

Project	Details	Repository
Alpaca	Code and documentation to train Stanford's Alpaca models, and generate the data.
BELLE	A 7B Large Language Model fine-tune by 34B Chinese Character Corpus, based on LLaMA and Alpaca.
Bloom	BigScience Large Open-science Open-access Multilingual Language Model
dolly	Databricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform
Falcon 40B	Falcon-40B-Instruct is a 40B parameters causal decoder-only model built by TII based on Falcon-40B and finetuned on a mixture of Baize. It is made available under the Apache 2.0 license.
FastChat (Vicuna)	An open platform for training, serving, and evaluating large language models. Release repo for Vicuna and FastChat-T5.
Gemma	Gemma is a family of lightweight, open models built from the research and technology that Google used to create the Gemini models.
GLM-6B (ChatGLM)	An Open Bilingual Pre-Trained Model, quantization of ChatGLM-130B, can run on consumer-level GPUs.
ChatGLM2-6B	ChatGLM2-6B is the second-generation version of the open-source bilingual (Chinese-English) chat model ChatGLM-6B.
GLM-130B (ChatGLM)	An Open Bilingual Pre-Trained Model (ICLR 2023)
GPT-NeoX	An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.
Jebadiah	Open decision models that return a probability for each option instead of generating text; available in 4B, 9B and 27B sizes.
Luotuo	A Chinese LLM, Based on LLaMA and fine tune by Stanford Alpaca, Alpaca LoRA, Japanese-Alpaca-LoRA.
Mixtral-8x7B-v0.1	The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts.
StableLM	StableLM: Stability AI Language Models

⬆ back to ToC

CV Foundation Model

Project	Details	Repository
disco-diffusion	A frankensteinian amalgamation of notebooks, models and techniques for the generation of AI Art and Animations.
midjourney	Midjourney is an independent research lab exploring new mediums of thought and expanding the imaginative powers of the human species.
segment-anything (SAM)	produces high quality object masks from input prompts such as points or boxes, and it can be used to generate masks for all objects in an image.
stable-diffusion	A latent text-to-image diffusion model

⬆ back to ToC

Audio Foundation Model

Project	Details	Repository
bark	Bark is a transformer-based text-to-audio model created by Suno. Bark can generate highly realistic, multilingual speech as well as other audio - including music, background noise and simple sound effects.
FunASR	Speech recognition toolkit with pretrained models and tools for voice activity detection, punctuation restoration, and speaker diarization.
whisper	Robust Speech Recognition via Large-Scale Weak Supervision

⬆ back to ToC

Robotics Foundation Model

[!NOTE] Emerging Architectures in VLA:

• Continuous Diffusion Language Models: Integrate diffusion heads or flow-matching to VLMs (e.g., DiVLA, OpenPI), enabling smooth, precise continuous action generation rather than discretized tokens.
• Recurrent Language Models: Utilize State Space Models (SSMs) like Mamba or recurrent transformers (e.g., RoboMamba, RD-VLA) to reduce inference memory and handle temporal dependencies, allowing iterative reasoning for complex robotic decision-making.

Project	Details	Repository
DiVLA	A continuous diffusion-based Vision-Language-Action model that integrates diffusion policies into autoregressive VLMs for robust and precise continuous robotic control.
LeRobot	A central community library by Hugging Face for AI in robotics — end-to-end learning tools, data pipelines, and support for training/deploying VLA models.
Octo	A transformer-based generalist robot policy pretrained on 800K+ robot trajectories from the Open X-Embodiment dataset. Supports language instructions, goal images, and fine-tuning to new embodiments.
OpenPI	Open-source VLA models from Physical Intelligence, including π₀ and π₀.5 — flow-based vision-language-action models pretrained on large-scale robot data with fine-tuning support.
OpenVLA	A 7B-parameter open-source Vision-Language-Action model trained on 970K+ robot demonstrations from the Open X-Embodiment dataset for generalist robotic manipulation.
RoboMamba	An efficient VLA model leveraging State Space Models (Mamba) instead of standard self-attention, offering linear inference complexity for efficient, recurrent robotic reasoning.
SmolVLA	A compact ~450M parameter VLA by Hugging Face, designed to be computationally efficient and accessible, running on consumer GPUs or CPUs. Part of the LeRobot ecosystem.

Serving

Large Model Serving

Project	Details	Repository
Alpaca-LoRA-Serve	Alpaca-LoRA as Chatbot service
Higgs	Rust inference server for Apple Silicon: MLX models behind OpenAI and Anthropic APIs, routing to remote providers, desktop dashboard.
OneComp	Fujitsu Research's post-training quantization pipeline for LLMs (QEP, AutoBit, JointQ, rotation) with vLLM plugin (arXiv:2603.28845).
CTranslate2	fast inference engine for Transformer models in C++
Clip-as-a-service	serving the OpenAI CLIP model
DeepSpeed-MII	MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
Faster Whisper	fast inference engine for whisper in C++ using CTranslate2.
FlexGen	Running large language models on a single GPU for throughput-oriented scenarios. (Archived)
Flowise	Drag & drop UI to build your customized LLM flow using LangchainJS.
lilbee	Single-binary local manager and search engine that stands up a llama-server fleet sized to your GPUs by gguf-parser for VRAM-aware multi-GPU placement
llama.cpp	Port of Facebook's LLaMA model in C/C++
LLMKube	Kubernetes operator for LLM inference with pluggable runtimes (llama.cpp, PersonaPlex/Moshi, generic), multi-GPU sharding, NVIDIA CUDA and Apple Silicon Metal support, and GGUF/MLX/SafeTensors model formats.
Shimmy	Python-free Rust inference server with OpenAI API compatibility and hot model swapping
Infinity	Rest API server for serving text-embeddings
Modelz-LLM	OpenAI compatible API for LLMs and embeddings (LLaMA, Vicuna, ChatGLM and many others)
NobodyWho	On-device LLM inference engine (Rust / llama.cpp) to embed local models directly in games and apps, with Godot / Flutter / React Native / Swift bindings; streaming, embeddings, tool calling, GBNF-structured output, and STT/TTS.
Off Grid	Open-source iOS/Android app running LLMs on-device via llama.cpp. Voice (Whisper), vision, image gen, tool calling — fully offline.
Ollama	Serve Llama 2 and other large language models locally from command line or through a browser interface.
Rapid-MLX	OpenAI-compatible LLM inference server for Apple Silicon using MLX. 2-4x faster than Ollama with tool calling and prompt caching.
TensorRT-LLM	Inference engine for TensorRT on Nvidia GPUs
text-generation-inference	Large Language Model Text Generation Inference
text-embeddings-inference	Inference for text-embedding models
tokenizers	💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
vllm	A high-throughput and memory-efficient inference and serving engine for LLMs.
whisper-ctranslate2	is a 4x faster and low-memory usage drop-in cli replacement that supports word-level timestamps and VAD filter
whisper.cpp	Port of OpenAI's Whisper model in C/C++
x-stable-diffusion	Real-time inference for Stable Diffusion - 0.88s latency. Covers AITemplate, nvFuser, TensorRT, FlashAttention. (Archived)

⬆ back to ToC

Frameworks/Servers for Serving

Project	Details	Repository
BentoML	The Unified Model Serving Framework
Bifrost	High-performance AI gateway written in Go. Unifies 23+ providers behind one OpenAI-compatible endpoint with automatic failover, adaptive load balancing, semantic caching, virtual keys and budgets, MCP tool calling, and native Prometheus observability.
Jina	Build multimodal AI services via cloud native technologies · Model Serving · Generative AI · Neural Search · Cloud Native
Mosec	A machine learning model serving framework with dynamic batching and pipelined stages, provides an easy-to-use Python interface.
mcpproxy-go	Open-source MCP proxy with BM25 tool filtering, quarantine security, activity logging, and web UI. Routes multiple MCP servers through single endpoint, reducing context bloat by ~97%.
SentryNode Gateway	Open-core AI inference governance and FinOps platform providing semantic routing, <40ms agent loop intervention, strict budget caps, and cryptographic audit trails. Includes voice pipeline governance (STT→LLM→TTS).
TFServing	A flexible, high-performance serving system for machine learning models.
Torchserve	Serve, optimize and scale PyTorch models in production (Archived)
Triton Server (TRTIS)	The Triton Inference Server provides an optimized cloud and edge inferencing solution.
langchain-serve	Serverless LLM apps on Production with Jina AI Cloud (Archived)
lanarky	FastAPI framework to build production-grade LLM applications
ray-llm	LLMs on Ray - RayLLM (Archived)
SIE	Open-source inference server and production cluster for serving embeddings, reranking, OCR, structured output, content safety, and generative models through one API.
Xinference	Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the cloud, on-premises, or even on your laptop.
KubeAI	Deploy and scale machine learning models on Kubernetes. Built for LLMs, embeddings, and speech-to-text.
Kaito	A Kubernetes operator that simplifies serving and tuning large AI models (e.g. Falcon or phi-3) using container images and GPU auto-provisioning. Includes an OpenAI-compatible server for inference and preset configurations for popular runtimes such as vLLM and transformers.
Open Responses	Serverless open-source platform for building long-running LLM agents with tool use.
KubeStellar Console	AI-powered multi-cluster Kubernetes dashboard for hybrid edge and cloud. GPU monitoring, LLM inference cluster management, benchmark streaming, and 20+ CNCF integrations. CNCF Sandbox (Apache 2.0).
laya-decision-api	FastAPI HTTP service for the local Laya System-1 decision model: offline structured decisions (choice / score / yes-no) with probabilities and confidence; MLX and torch backends

⬆ back to ToC

Security

Frameworks for LLM security

Project	Details	Repository
API Relay Audit	Local security audit for AI API relays and LLM proxies; checks prompt injection, model identity drift, tool-call rewriting, error leakage, SSE anomalies, and Web3 wallet probes.
Cordum	Safety-first agent orchestration platform with pre-dispatch policy evaluation, output scanning (PII, secrets, injection), job scheduling, workflow engine, and full audit trail.
brood-box	CLI tool for running coding agents inside hardware-isolated microVMs with snapshot isolation, egress control, and MCP authorization.
DOS Kernel	Deterministic trust kernel for AI-agent fleets: verifies agent "done" claims from git evidence rather than self-report, arbitrates concurrent file access between agents, and audits commit messages against their own diffs.
dstack	Open-source confidential AI framework for secure LLM deployment with data privacy, providing hardware-enforced isolation using Intel TDX and NVIDIA Confidential Computing.
EmbedGuard	Cross-layer detection and provenance attestation for adversarial embedding attacks in RAG systems. Flags poisoned or manipulated embeddings before they reach the retrieval layer.
Plexiglass	A Python Machine Learning Pentesting Toolbox for Adversarial Attacks. Works with LLMs, DNNs, and other machine learning algorithms.
PrivacyScrubber	Zero-Trust client-side PII and developer secrets sanitizer and MCP server for Cursor, Windsurf, and Claude Desktop with sub-2ms in-memory tokenization.
PrivAiTe	Self-hosted OpenAI-compatible proxy that reversibly pseudonymizes PII before requests reach the provider, tool-call arguments and multimodal included.
shim	Open-core AI gateway that redacts PII before prompts reach model providers (Presidio plus Turkish TCKN/VKN recognizers). The source-available enterprise edition adds a tamper-evident hash-chain audit log with daily Merkle anchors as evidence for EU AI Act and KVKK record-keeping.
wardcat	On-premises PII and sensitive-data detection and masking for LLM and RAG workflows, combining regex, NER, and optional local LLMs with reversible masking.

⬆ back to ToC

Observability

Project	Details	Repository
AgentCost	Track and optimize LLM API costs across OpenAI, Anthropic, and LangChain.
Azure OpenAI Logger	"Batteries included" logging solution for your Azure OpenAI instance.
budget-guard	Zero-infra hard spending caps for LLM APIs — blocks before overspending (atomic reservation), per-feature cost attribution, streaming usage, retry-storm detection, and adapters for Vercel AI SDK / LangChain.js / LlamaIndex.TS / Mastra.
ClevAgent	Runtime monitoring for AI agents — heartbeat watchdog, loop detection, cost tracking, auto-restart. Python SDK or HTTP API.
CodeBurn	Local-first cost and token tracking for AI coding agents (Claude Code, Codex, Cursor and 38 more) by project, model and task, with waste detection and plan quota tracking.
Deepchecks	Tests for Continuous Validation of ML Models & Data. Deepchecks is a Python package for comprehensively validating your machine learning models and data with minimal effort.
digline	Regression testing for LLM apps: approved scores, prompt and commit pinned as a versioned baseline in your repo; compare says which case got worse and by how much.
Evidently	An open-source framework to evaluate, test and monitor ML and LLM-powered systems.
EvalView	Regression testing for AI agents. Snapshot behavior, detect tool-call and output regressions, with golden-baseline diffing and LLM-as-judge scoring. Supports LangGraph, CrewAI, OpenAI, Claude, and any HTTP API.
Fiddler AI	Evaluate, monitor, analyze, and improve machine learning and generative models from pre-production to production. Ship more ML and LLMs into production, and monitor ML and LLM metrics like hallucination, PII, and toxicity.
Flowlines	Behavioral observability for production AI agents that detects drift, context loss, constraint violations, and user frustration across sessions.
Giskard	Testing framework dedicated to ML models, from tabular to LLMs. Detect risks of biases, performance issues and errors in 4 lines of code.
QWED	Deterministic verification protocol for LLM outputs using 8 formal verification engines (SymPy, Z3, AST, SQLGlot). Prevents hallucinations through mathematical proofs rather than statistical methods.
Great Expectations	Always know what to expect from your data.
Helicone	Open source LLM observability platform. One line of code to monitor, evaluate, and experiment with features like prompt management, agent tracing, and evaluations.
hermes-rubric	Evidence-first assessment for agent outputs and applications. Synthesizes, loads, or accepts a rubric; collects citations for each dimension; scores against accepted evidence; reports coverage; and emits reproducibility receipts. Apache-2.0.
Inferrail	Per-run dollar budgets for AI agents and jobs: an OpenAI- and Anthropic-compatible gateway (in-process or self-hosted) that reserves each call's worst-case cost before it is sent, refuses calls that don't fit before the provider, and keeps a payload-free cost record per run.
JevOss	Test suite for decision-model servers that implement the Jev API: accuracy, probability calibration, and probes for option order, injected instructions, irrelevant padding, yes/no rewording and repeated calls. Apache-2.0.
Traceloop OpenLLMetry	OpenTelemetry-based observability and monitoring for LLM and agents workflows.
Langfuse 🪢	Open-source LLM observability platform that helps teams collaboratively debug, analyze, and iterate on their LLM applications.
Lookspan	Local-first observability for AI agents. One command (npx lookspan) runs a dashboard with traces, a timeline/waterfall view, cost tracking, replay & LLM-as-judge, and datasets. MCP-native, with OpenAI/Anthropic drop-ins and an OpenTelemetry receiver — data stays on your machine.
whylogs	The open standard for data logging
Maxim AI	Platform for AI Agent Simulation, Evaluation & Observability
Noveum	AI agent tracing, evaluation datasets from production interactions, and model comparisons on quality, latency, and cost.
onWatch	Lightweight Go CLI that tracks AI API quota usage across 7 providers (Anthropic, OpenAI, GitHub Copilot, MiniMax, and more). Background daemon, <50MB RAM, zero telemetry, SQLite storage.
OpenGATE	Deterministic verification for evidence-grounded AI — gold-anchored grounding checks (required facts, ungrounded numbers, abstention) with no LLM judge, plus CI regression gates, pytest/DeepEval integrations, and an MCP server.
otelcol-genai-sketches	OpenTelemetry Collector connector that tracks LLM token usage by user and session without a Prometheus cardinality explosion: bounded per-model metrics plus keyed top users and sessions. Includes a tested LiteLLM recipe and a Helm chart.
phoenix2pytest	Reads Arize Phoenix traces flagged as failures and synthesises pytest regression tests, so production-discovered LLM failures feed back into the test suite without manual translation.
RAGeval	Drop-in LLMOps observability platform for RAG with built-in UI and tracing.
RagTune	CLI tool for debugging and benchmarking RAG retrieval. EXPLAIN ANALYZE for your retrieval layer.
runtape	Finds which part of an agent's context caused a bad decision by rerunning it with pieces removed, checks candidate fixes the same way, and writes a pytest regression test. Runs locally with OpenAI, Anthropic, LangGraph and Ollama.
traceAI	Open-source AI tracing framework built on OpenTelemetry for deep observability across agentic and LLM workflows.
Future AGI	Production-grade SDK for observability, automated evaluations and prompt management with sub-100ms guardrails for LLM/agent workflows.
semantic-coverage	Visualizes RAG knowledge gaps and "blind spots" using 2D UMAP clustering and density detection.
Token Police	SDK (Python, Node) that prices each LLM call before it is made and blocks or reroutes it on per-user, per-session or per-workflow budget rules; prompt text never leaves the app.
Weco Observe	Observability and debugging tool for AI research agents. Trace multi-step LLM agent runs, visualize decision trees, and identify failure modes in autonomous research workflows. Cloud hosted with open-source agent integration.
witness	A recording cache for model APIs: one Rust binary that proxies any LLM endpoint, dedupes repeat calls, and writes every model call, MCP tool call, and command side effect into a tamper-evident hash-chained journal with Ed25519 agent identity and Merkle proofs anchored to Sigstore Rekor.
wobbly	Metamorphic testing for LLM extraction — catch wrong outputs with no ground-truth labels: perturb inputs in ways that shouldn't change the answer, flag when they do.

⬆ back to ToC

LLMOps

Project	Details	Repository
agenta	The LLMOps platform to build robust LLM apps. Easily experiment and evaluate different prompts, models, and workflows to build robust apps.
agent-convergence-scorer	Zero-dependency Python CLI and library for measuring lexical convergence, exact-match rate, overlap, and divergence across parallel agent runs.
AgentMark	Type-Safe Markdown-based Agents
AgentField	Open-source control plane for building and operating AI agents like APIs at scale, with routing, memory, observability, identity, auth, and policy controls.
Agnos	Gateway-agnostic, self-hosted LLM control plane (MIT): an OpenAI-compatible governance proxy (auth, budgets, guardrails, cost, observability) that runs LiteLLM, Bifrost or Portkey as a swappable, stateless translation engine, so provider keys stay in your own control plane instead of the gateway.
AI studio	A Reliable Open Source AI studio to build core infrastructure stack for your LLM Applications. It allows you to gain visibility, make your application reliable, and prepare it for production with features such as caching, rate limiting, exponential retry, model fallback, and more.
AISIX	Open-source AI g