大模型推理服务

Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/

Expose your FastAPI endpoints as Model Context Protocol (MCP) tools, with Auth!

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

Brigade — Your personal intelligence, built enterprise-grade

Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.

Large Language Model Text Generation Inference

Official inference library for Mistral models

The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.

PyTorch implementation of YOLOv3, YOLOv3-SPP, and YOLOv3-tiny for real-time object detection with training, validation, inference, and multi-format export.

High-speed Large Language Model Serving for Local Deployment

Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source, speech, and multimodal models on cloud, on-prem, or your laptop — all through one unified, production-ready inference API.

ChatRWKV is like ChatGPT but powered by RWKV (100% RNN) language model, and open source.

Anomaly detection related books, papers, videos, and toolboxes. Last update late 2025 for LLM and VLM works!

🧑‍🚀 全世界最好的LLM资料总结（多模态生成、Agent、辅助编程、AI审稿、数据处理、模型训练、模型推理、o1 模型、MCP、小语言模型、视觉语言模型） | Summary of the world's best LLM resources.

Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.

Accessible large language models via k-bit quantization for PyTorch.

Run frontier LLMs and VLMs locally on Qualcomm devices across NPU, GPU, and CPU with a few lines of code

Put an end to code hallucinations! GitMCP is a free, open-source, remote MCP server for any GitHub project

Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.

The chat decision assistant: before you reply, Jev reads the chat, judges intent and risk, and drafts replies you fill in with one tap. You press send. Android · Windows · macOS · iOS | 聊天辅助决策工具：先看懂对方再回复，发不发由你。

Semantic cache for LLMs. Fully integrated with LangChain and llama_index.

Own your AI. The native macOS harness for AI agents -- any model, persistent memory, autonomous execution, cryptographic identity. Built in Swift. Fully offline. Open source.

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

The official repository for ERNIE 4.5 and ERNIEKit – its industrial-grade development toolkit based on PaddlePaddle.

Open-source observability for your GenAI or LLM application, based on OpenTelemetry

File Parser optimised for LLM Ingestion with no loss 🧠 Parse PDFs, Docx, PPTx in a format that is ideal for LLMs.

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

🔥 基于大模型和 RAG 的智能问数系统，对话式数据分析神器。Text-to-SQL Generation via LLMs using RAG.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

FlashInfer: Kernel Library for LLM Serving

Open-source framework for building agentic apps in JavaScript, Go, Dart, and Python, built and used in production by Google

A RWKV management and startup tool, full automation, only 8MB. And provides an interface compatible with the OpenAI API. RWKV is a large language model that is fully open source and available for commercial use.

On-device Speech AI for Apple Silicon

Towards Human-Sounding Speech

Generate audiobooks from EPUBs, PDFs and text with synchronized captions.

2bit quants, kernels and runtime for consumer devices.

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

Solve Visual Understanding with Reinforced VLMs

Enchanted is iOS and macOS app for chatting with private self hosted language models such as Llama2, Mistral or Vicuna using Ollama.

QOwnNotes is a plain-text file notepad and todo-list manager with Markdown support and Nextcloud / ownCloud integration.

🐢 Open-Source Evaluation & Testing library for LLM Agents

Open-source alternative to Opus Clip, Vidyo.ai, Klap & SubMagic. Turn long-form YouTube videos into viral 9:16 shorts using LLM highlight detection, Whisper transcription, and auto vertical cropping — free, no watermarks, no per-clip credits.

MLX-VLM is a package for inference and fine-tuning of Vision Language Models (VLMs) on your Mac using MLX.

AI-powered virtual executive team — a single coherent executive persona backed by 8 specialist agents (FastAPI + Next.js).

learn LLM inference system on Apple Silicon for systems engineers: build a tiny vLLM + Qwen

Robust recipes to align language models with human and AI preferences

Turn any script into viral, multi-style animated stickman videos for Faceless YouTube Channels. Automate visuals, voiceover, captions & SFX. 100% Free & Open-Source.

The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices

Superduper: End-to-end framework for building custom AI applications and agents.