zero-to-sglang

---

Course status: Part 0, Part I, Chapters 1–5 and 8 of Part II, and Chapters 1 and 3 of Part III (Attention Backends and Hierarchical Caching) are complete in both English and Chinese. The course is still under development. You can switch to 简体中文 at any time. Want to help? See Contributing.

🎯 About

  Large models have moved on from the training race to the inference race. Once applications took off, inference cost and latency became the real bottleneck: on the same GPU, a good inference engine can serve several times more requests. Yet there are few tutorials on inference engines. They either stay at the concept level without touching code, or drop you straight into the SGLang source with no idea where to start.

  This course fills that gap. The first half starts from the core questions of inference: why KV Cache exists, how prefill differs from decode, what compute-bound and memory-bound actually mean. The middle part walks you through building a mini-sglang from scratch: forward pass, generation, KV Cache, HTTP serving, Continuous Batching, Paged KV Cache, RadixAttention, added one at a time. The last part returns to the real SGLang, explains how its cutting-edge optimizations are actually implemented, and finally shows you how to land your first solid PR in SGLang.

  The project is jointly launched by Datawhale and RadixArk (the company founded by the SGLang team). SGLang is technically solid and actively iterated in the inference framework space. If you like this project, please give the official SGLang repository a ⭐.

✨ What you will learn

• 📖 Understand inference: KV Cache, prefill/decode, compute-bound vs. memory-bound, and what these concepts actually mean
• 🏗️ Build the engine: write a mini-sglang from 0 to 1, with every step landing in code
• 🛠️ Read the real engine: how RadixAttention, Paged KV Cache and Continuous Batching are implemented inside SGLang
• ⚙️ Touch the frontier: quantization, hierarchical caching, DP Attention / EP / PP, Prefill-Decode disaggregation
• 🚀 Join open source: learn profiling and trace analysis, and walk through the full SGLang PR workflow

📋 Prerequisites

• Python: fluent in Python, with basic software engineering habits
• Deep learning basics: familiar with PyTorch and the fundamentals of neural networks
• Math: linear algebra and probability; knowing roughly how matrix multiplication and attention are computed is enough
• GPU programming (optional): knowing CUDA basics helps, but is not required. Part I fills in the gaps from scratch
• Hardware (optional): Part I needs no GPU; most of Part II can be debugged on CPU, but the full implementation and performance tests are best done on a GPU (cloud is fine)

🔗 Links

• Repository: https://github.com/datawhalechina/zero-to-sglang
• SGLang: https://github.com/sgl-project/sglang
• mini-sglang reference implementation: https://github.com/sgl-project/mini-sglang

📖 Syllabus

Status legend: ✅ Done 🔄 In progress 📝 Needs polish 🚧 Planned ⏸️ Paused

Chapters in bold are written by SGLang core members.

Chapter	Key content	Status
Part 0 — Before you learn
0.1 Coding Ethics and Open-Source Spirit	Own your code, communicate like a human, profile first, the open-source spirit	✅
0.2 Deploy Your First SGLang Server	Environment setup; run Qwen3-0.6B with SGLang on your own GPU	✅
Part I — Foundations (concepts only, no code, no GPU)
1. Introduction to LLM	What an LLM is and how it evolved, the Transformer architecture, autoregressive generation, key concepts	✅
2. Introduction to Inference	Training vs. inference, prefill/decode, compute-bound vs. memory-bound, the Roofline model	✅
3. Introduction to GPU	GPU architecture basics, how LLM inference executes on a GPU, understanding inference bottlenecks from the hardware	✅
4. KV Cache: The Core Data Structure of Inference	Deriving KV Cache from attention, cache lifecycle, quantitative memory analysis	✅
5. Introduction to Benchmark	Core metrics such as TTFT / TPOT / ITL / Goodput, percentiles and tail latency, how to design, run and read a benchmark	✅
Part II — Build Your Own Mini SGL
1. mini-sglang: What an Inference Engine Looks Like	Overall architecture of an inference engine, module breakdown, roadmap for this part	✅
2. Inside SGLang: The Path of a Request (Concepts) · Code Walkthrough	The full lifecycle of a request from arrival to response	✅
3. Your First 200 Lines: Forward Pass and Generation (Concepts) · Code Walkthrough	A source walkthrough of embeddings, attention, the LM head, and generation	✅
4. KV Cache: From O(n²) to O(n) (Concepts) · Code Walkthrough	Implementing the cache and optimizing attention computation	✅
5. HTTP and Concurrent Requests (Concepts) · Code Walkthrough	HTTP basics and request flow; implementing isolation, streaming, and cancellation	✅
6. Continuous Batching and the Scheduler (Concepts) · Code Walkthrough	Continuous batching, scheduler state, resource limits, and latency/throughput tradeoffs	✅
7. Paged KV Cache and Memory Management (Concepts)	Paged KV Cache and GPU memory management	✅
8. RadixAttention and Prefix Caching (Concepts) · Code Walkthrough	RadixAttention and prefix caching	✅
9. Multi-process & Tensor Parallelism(Concepts) · Code Walkthrough	Multi-process execution and tensor parallelism	✅
10. Speculative Decoding (Concepts) · Code Walkthrough	Drafting and verification, greedy and sampling-based acceptance, rejection correction, and performance tradeoffs	✅
Part III — Advanced Inference Technique (deep into the real SGLang)
1. Attention Backends (FlashInfer / Triton / FA3 / FlashMLA)	Backend responsibilities, tiled Attention, implementation tradeoffs, and selection	✅
2. Quantization and Low-Precision Inference	Numeric formats and scales, Decode speedup and KV Cache capacity, accuracy tradeoffs, quantization labels and loading	✅
3. Hierarchical Caching	L1/L2/L3 cache tiers, prefix matching, backup and restoration, prefetching and performance tradeoffs	✅
4. Scaling Out: DP Attention, EP, PP	Scaling out: DP Attention / EP / PP	🚧
5. Prefill-Decode Disaggregation	Prefill-Decode disaggregation	🚧
Part IV — How to Make Contribution to SGLang (optional)
1. Deploying SGLang with the Cookbook	Deploying SGLang with the Cookbook	🚧
2. Measuring It All: Profiling and Trace Analysis	Profiling and trace analysis	🚧
3. SGLang PR Workflow	The full SGLang PR workflow	🚧

🗓️ Writing schedule

Start date: Monday, August 24, 2026. About one week per chapter; chapters in bold are written by SGLang core members.

Part	Schedule	Status
Part 0 — Before you learn	Done	✅
Part I — Foundations	Writing 8.24 ~ 9.06, review 9.07 ~ 9.13	✅
Part II — Build Your Own Mini SGL	Writing 9.14 ~ 10.04, review 10.05 ~ 10.10	🔄
Part III — Advanced Inference Technique	10.10 ~ 11.15	🔄
Part IV — How to Make Contribution to SGLang	11.16 ~ 12.6	🚧

🚀 Quick start

# Clone the repository
git clone https://github.com/datawhalechina/zero-to-sglang.git
cd zero-to-sglang
# Install dependencies as each chapter requires

Learning path

1️⃣ Read Part I to build an overall picture of inference (no GPU needed)
2️⃣ Follow Part II to build a mini-sglang from 0 to 1
3️⃣ Move to Part III and understand the frontier optimizations against the real SGLang source
4️⃣ Finish Part IV: profiling, trace analysis, and the full PR workflow

Project structure

zero-to-sglang/
├── course-material/         # Course text
│   ├── ch/                  # Chinese edition (read online at /ch/)
│   │   ├── part0/           # Part 0: Before you learn
│   │   ├── part1/           # Part I: Foundations (complete)
│   │   ├── part2/           # Part II: Build Your Own Mini SGL (in progress)
│   │   ├── part3/           # Part III: Advanced Inference Technique (planned)
│   │   ├── part4/           # Part IV: How to Make Contribution to SGLang (planned)
│   │   └── WRITING_TEMPLATE.md
│   └── eng/                 # English edition (read online at /eng/, in progress)
│       ├── part0/
│       └── WRITING_TEMPLATE.md
├── community/               # Community corner: build logs, pitfalls, study notes
│   ├── ch/
│   └── eng/
├── docs/.vitepress/         # VitePress site config
├── README.md                # Project overview (English, default)
├── README_zh.md             # Project overview (Chinese)
├── README_en.md             # Legacy English entry (links to README.md)
└── .gitignore               # Git ignore rules

🤝 Contributing

We are an open community and welcome contributions of every kind. Before you start, please read Part 0 to understand the coding ethics and open-source spirit we expect.

1. Improve the course text

• 🐛 Found an error in the content, a formula, or code: open an issue, or fix it and open a PR
• 📝 Add to a chapter or improve the wording: open a PR. Chapter division and numbering follow the outline in the writing template; do not add, remove, or merge chapters on your own
• 🌐 Translate: help translate chapters into English following the English writing template, and put them under course-material/eng/
• 💡 Have ideas about the course: open an issue to discuss

2. Share your practice

• 🔧 Built mini-sglang yourself following Part II? Write down the bugs you hit and how you fixed them
• ⚠️ Pitfalls from deploying SGLang or setting up the environment, or places where the docs were unclear and others deserve a warning
• 📒 Study notes written in your own words after finishing a chapter, or extra derivations and experiments

Community content goes under the matching directory in community/eng/ (English) or community/ch/ (Chinese).

How to open a PR

1. Fork the repository and create a branch from main
2. PR requirements are in community/eng/PR_requirement.md; check every item before submitting

❓ FAQ

Yes. Part I involves no GPU at all. Most of the code in Part II can be debugged on CPU, but full performance verification and the deeper content of Part III are best done on a GPU (cloud is fine).

No. The course starts from inference concepts. Low-level topics such as attention backends only come in Part III, and you can catch up on CUDA as needed at that point.

💬 Reader Community

Join the zero-to-sglang reader community to learn together, ask questions, and share ideas: Join our Discord community

👥 Contributors

We thank every developer who has contributed to this project!

Special thanks

• Thanks to Datawhale and the SGLang team for supporting the project
  • Datawhale: @1iyouzhen, @xuhu0115, @kangkang-Adam
  • SGLang: @Ccyest, @fcranzhou
• Thanks to @Sm1les for the help and support
• Thanks to every developer who has contributed to this project ❤️

🎓 Citation

If zero-to-sglang helps your research or work, please cite it:

@misc{zero_to_sglang2026,
  title  = {zero-to-sglang: Building an LLM Inference Engine from Scratch},
  author = {TODO},
  year   = {2026},
  url    = {https://github.com/datawhalechina/zero-to-sglang},
  note   = {GitHub repository}
}

⭐ Star History

If this project helps you, please give it a Star ⭐️!

📄 License

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

---