cs249r_book

Machine Learning Systems

---

Why This Curriculum Exists

AI engineering is the discipline of building efficient, reliable, and safe intelligent systems that operate in the real world, not models in isolation. It deserves a place alongside software engineering and computer engineering, and this curriculum teaches it the way computer architecture has long been taught, from first principles and with numbers a student can recompute.

Those numbers come from constraints. A model that scores well on a benchmark still has to fit in memory, meet a latency budget, stay within a power limit, and fail safely in front of real people. These constraints decide the architecture, and they follow laws that outlast any framework or chip. Memory bandwidth, network latency, energy, and failure rates govern a microcontroller with 256 KB of memory and an accelerator in a datacenter alike, even though the two differ in scale by many orders of magnitude. I learned this while working on TinyML at Google, and it became the idea that runs through every volume: constraints drive architecture.

Teaching AI engineering this way takes more than a book, which is why everything lives in one repository. Each part teaches something the others cannot. Students who only read do not internalize, and students who only code do not generalize. The textbook builds the mental model, TinyTorch makes it concrete by having students build the machinery themselves, and the hardware kits and the MLSys·im simulator show where the model breaks, on devices students can hold and at scales beyond any one lab.

The work began in a classroom. The HarvardX TinyML course came first and reached more than 100,000 learners. In 2023, the students in CS249r at Harvard wrote the first class notes for this curriculum together, one topic per group, and we published them on GitHub. Since then, I have kept revising the material with help from a growing community of contributors, and instructors at universities around the world now teach from it. The goal is to reach one million learners by 2030, because the world is building AI factories faster than it is training AI engineers.

---

What You Will Learn

Each chapter connects an idea a reader may already know from machine learning to the systems that make it work in practice.

---

The Four Volumes

The textbook spans four volumes. Each one widens the boundary of the system and raises the cost of failure, so the volumes differ in scope, not in depth.

Each volume answers a question the previous one leaves open.

• Vol I → Vol II, when one machine is not enough. Volume I covers single-machine execution, memory walls, and kernel efficiency. Volume II carries those foundations across thousands of accelerators, collective communication, fault-tolerant networks, and datacenter orchestration.
• Vol II → Vol III, when a stateless response is not enough. Volume II serves stateless requests at scale. Volume III adds stateful, multi-step loops that manage context memory, call tools, run in sandboxes, and recover from nondeterministic failures.
• Vol III → Vol IV, when software acts on the physical world. Volume III governs software environments. Volume IV crosses into physical systems, where computational latency becomes distance traveled and a failure cannot be undone.

I am writing Volumes III and IV because I plan to teach both subjects, and because they are where machine learning systems are heading. The progression is a natural one. Volumes I and II run in an open loop, where a request goes in and a response comes out. Volume III closes the loop in software, where an agent acts, observes the result, and acts again. Volume IV closes it through the physical world. Writing a book is how I learn a subject well enough to teach it.

⚠️ Volumes III and IV are open drafts. Chapters are added, reorganized, and rewritten as my understanding grows, so please do not cite or teach from them yet. Feedback is welcome through the book feedback issue forms.

---

What's in the Repository

Around the textbook sit the companion tools that let students build, measure, and practice what they read. The pieces are designed to work together, but none of them requires the others.

Two earlier-stage experiments also live here. SocratiQ is an AI tutor that adds guided reading, quizzes, and spaced repetition to static learning sites. MLPerf EDU is a teaching benchmark suite aligned with MLCommons MLPerf.

---

Start Here

Where to begin depends on what you want from the curriculum.

---

How the Material Is Made

No chapter is written once. Each one goes through a cycle that repeats as the field changes.

1. Watch the field. I follow the research and industry venues where new work appears: ISCA, MICRO, HPCA, ASPLOS, ISPASS, MLSys, NeurIPS, ICML, ICLR, Hot Chips, and industry summits.
2. Distill. A chapter keeps the principles and leaves out the variations. Quantization, for example, has been implemented in dozens of ways, and a student needs the principles that explain all of them. To decide what belongs, I use a test that David Patterson, who wrote the foreword to Volume I, gave me. A technique earns a place only when several companies use it in practice. That test ties the material to systems in production, so what a student learns carries over to practice.
3. Review with practitioners. I review chapters one-on-one with the engineers who build these systems inside large companies, from hyperscale infrastructure engineers to model developers. Practitioners at Arm, Google, Intel, Meta, Microsoft, Qualcomm, STMicroelectronics, and other organizations have reviewed the material, and academic reviewers in Europe, Asia, and the Americas have tested it against their own research and teaching.
4. Teach it. Volume I has been taught at Harvard through CS249r since 2023 and adopted by instructors at UC Berkeley, Columbia, Georgia Tech, IIT Bombay, and universities across Europe, Latin America, Africa, and Asia. The classroom shows where students get stuck.
5. Revise. Reader issues, reviewer notes, and classroom experience go back into the next draft.

Reviewers do more than check facts, and sometimes they change what the book covers. Sustainable AI and responsible AI both entered the book this way. Engineers building AI datacenters told me that the footprint of these facilities keeps growing and that students need to learn to account for it. Their concern led to a chapter I had not planned, which I wrote in 2023, before sustainable AI was a common topic.

Engineers in industry also urged me to teach responsible AI. It can look like a subject outside traditional engineering, but the systems in this book make decisions that affect people, and more and more of them act on their own. The engineers who build such systems are responsible for how they behave, so the book teaches that responsibility as part of the engineering.

Readers are part of the review, too. Every chapter is open, and when something is wrong, unclear, or missing, the book feedback forms send it straight into the next revision.

Numbers are computed, not typed. Every derived quantity in the books comes from Python code that ships with the text. I learned the need for this the hard way. In early drafts I typed numbers by hand, made mistakes, and watched the same quantity drift apart from one chapter and one volume to the next. No author can hold every number across the series in mind at once, so I wrote MLSys·im to hold them instead. It is the single source of truth for every chapter and every volume. Each hardware and model specification in it traces to a cited source, and when a value changes there, every chapter that uses it changes with it. A test suite runs on every commit and checks unit consistency, cross-references, notation, and citations.

---

FAQ

When I began teaching machine learning systems, there was no textbook for the course. Books covered the models, and papers covered individual systems, but nothing taught the infrastructure underneath them as one subject. The field was changing too quickly for anyone to have written one. So I started writing it myself, because I have always learned best by building from scratch and teaching others.

When I want to understand an emerging or complex systems area, writing about it and writing code for it force me to slow down. They force me to find the principles that last, work through the math, and separate what is fundamentally true about physics and hardware from what is merely current.

This repository is my open workshop and teaching lab. It is where I work out ideas for my students at Harvard, my research lab, and open benchmarking efforts like MLPerf. Volumes III and IV are that process happening in public.

There is also a deeper reason. In the late 1990s, as an undergraduate, I wrote to a professor at Stanford because I was fascinated by genetic algorithms. I did not expect an answer. Then a brown package arrived at my house with a VHS tape on genetic algorithms inside, sent from the professor's office. I no longer remember the professor's name, so I never got to say thank you, but I have not forgotten what it felt like to have someone I had never met take the time to share what they knew with a curious student. That package sparked my curiosity, and the feeling has stayed with me.

Students write to me now the way I once wrote to that professor, far more than I can answer one by one. This curriculum is my way of sending the package back out, to a much bigger classroom than any lecture hall can hold.

It is for anyone who wants to understand how intelligent systems run, not only how models are trained: students, software engineers moving into ML infrastructure, hardware architects, and educators designing courses.

The only assumptions are that the reader can program in Python and has met basic machine learning ideas, such as a neural network or a loss function. The systems concepts are built from the ground up. No prior background in computer architecture, networking, or distributed systems is needed.

No. The two volumes differ in scope, not in depth or rigor.

Volume I covers one machine with a handful of accelerators, from the memory hierarchy and kernel efficiency to quantization and on-device deployment. Volume II covers the fleet, with thousands of accelerators connected by high-speed networks, the parallelism that spans them, fault tolerance, and serving at datacenter scale.

Volume II does not assume that a reader has finished Volume I, so a reader with strong single-machine foundations can start there. Reading Volume I first gives the cleanest mental model from the silicon up, in the same way that Hennessy and Patterson's Computer Organization and Design precedes Computer Architecture: A Quantitative Approach.

Each volume stands on its own as a textbook. The companions are there for readers who want to go further. Reading builds the mental model, and building and measuring make it last. Start with the book and add the hands-on pieces when an idea needs to become practice.

Books such as Goodfellow, Bengio, and Courville's Deep Learning, Bishop's texts, d2l.ai, and fast.ai teach the mathematics, architectures, and optimization of models, and they teach them well.

This curriculum starts where those books stop. It treats the model as one component inside a system that must ingest data, run on real silicon under power and thermal limits, answer queries within milliseconds, and keep running across thousands of machines. A reader can finish an excellent deep learning course and still not know why a cluster idles at 30% utilization, why serving memory grows with the KV cache, or how gradients are synchronized without stalling the network. Those questions are the subject here.

MLOps books, such as Chip Huyen's Designing Machine Learning Systems, focus on operational pipelines: feature stores, workflow orchestration, and deployment with current industry tools. They are valuable practical guides.

This curriculum focuses on what lies underneath: the hardware limits, quantitative trade-offs, and first principles that explain why those tools are designed as they are, and the limits no tool escapes, such as memory bandwidth, network latency, energy, and failure rates.

The two differ the way a recipe differs from an understanding of how heat, salt, fat, and time interact. A recipe works until the oven changes or the batch grows tenfold. Understanding the chemistry lets a cook work in any kitchen, fix a dish that went wrong, and invent new ones.

Barroso, Hölzle, and Clidaras give a firsthand account of how one organization built hyperscale infrastructure, and I admire that work. This curriculum aims to complement it in three ways:

1. A progressive curriculum, not a reference. It starts from a single machine and its memory hierarchy (Volume I) and builds up to multi-node clusters and warehouse-scale fleets (Volume II), with quantitative exercises along the way.
2. Vendor-neutral foundations. It generalizes across accelerator architectures (GPUs, TPUs, custom ASICs) and deployment targets (cloud clusters to milliwatt microcontrollers), so the principles carry across hardware generations.
3. Hands-on and living. The text is surrounded by code readers build themselves (TinyTorch), open hardware kits, cluster modeling (MLSys·im), and community contributions, and it changes as the field does.

Yes, and I would rather say so plainly than leave readers to guess. I use generative AI tools to write code and the checks that test it, to survey the literature before I read the papers myself, to draft passages that I then rewrite, to help produce figures that I once drew by hand, and to audit the books for consistency. I use them on the companions as well, from TinyTorch and the labs to the slides and the StaffML question bank. They are also how I learn a new area quickly. The tools take on mechanical work, which leaves my time for the parts that need a teacher: what to include, how to explain it, and whether it is right.

A tool can write text that sounds right and is wrong, so I do not take its output on trust. I check the work in ways that do not depend on the tool. Every number in the books is computed by code, and automated checks run whenever something changes. Practitioners review the chapters, and students test them in class. In StaffML, each question records whether it began as a machine draft and whether a person has reviewed it.

I researched, wrote, and rewrote every chapter. The decisions about what goes in, what stays out, and how an idea is taught are mine, and so is every mistake. Because all of the work is open, anyone can check it and tell me what they find.

Volumes III and IV have not yet been through the classroom or practitioner review, which is why they are marked as drafts. Each volume includes a full Note on AI Assistance.

Language models are good at retrieval, code generation, and summarizing facts. A paragraph that only delivers a fact a reader could look up in seconds has not earned its place in a textbook.

What a textbook provides is perspective and a coherent mental model: the order in which to learn ideas, how trade-offs constrain one another, and what not to worry about. Deciding what to leave out is one of the most important choices an author makes.

Bruce Davie made this point in his essay “Textbooks in Tokenland” (Systems Approach). A language model generates text without communicative intent, he writes, while a textbook is a deliberate path from a teacher to a student toward a shared understanding of how the world works.

The two are not in opposition. The book provides the structure and the quantitative argument, and interactive tools let a reader test that understanding. The Marimo labs ask a reader to predict an outcome and then check it, and the SocratiQ tutor answers questions about the page being read.

Yes. Every volume is free to read online at mlsysbook.ai.

The textbook is released under a Creative Commons license (CC BY-NC-ND 4.0), the same license as the MIT Press print edition. It may be read and shared for non-commercial use with attribution. Translations and other modified versions need permission. The companion software is open source under its own licenses, listed under License.

---

Support This Work

If this curriculum has helped you, there are two simple ways to help it grow.

---

Contributing

Branches. Development happens on dev, which this page shows. The main branch holds what is live on mlsysbook.ai, and the publish workflow promotes dev to main at each release.

---

License

This is a multi-component repository, and each component is released under its own license to match its purpose. The license file inside each directory, such as tinytorch/LICENSE or staffml/app/LICENSE, is authoritative.

A shorter summary for readers is at mlsysbook.ai/about/license.

If you are an institution considering adoption, or a company interested in commercial terms for a copyleft component, please write to [email protected].

---

Acknowledgments

I assembled this curriculum, but I did not invent most of what it teaches. The ideas come from the engineers and practitioners in industry who build these systems every day, and from researchers worldwide who study them and publish what they learn. The books cite their work throughout, and the curriculum would not exist without it. My own research lab has contributed as well.

Many people also gave their time to the curriculum itself. My students in CS249r at Harvard have tested every revision. Reviewers in industry and academia read chapters and told me where I was wrong. Instructors who adopted the material reported what worked in their classrooms and what did not. Readers found errors, and contributors fixed them. The financial contributors on Open Collective helped carry the work to more students.

Whatever is good here owes a great deal to all of them. The mistakes are mine.

---

Contributors

These people have improved the textbooks, the software, and the teaching materials.

Legend: 🪲 Bug Hunter · 🧑‍💻 Code Contributor · ✍️ Doc Wizard · 🎨 Design Artist · 🧠 Idea Spark · 🔎 Code Reviewer · 🧪 Test Tinkerer · 🛠️ Tool Builder

📖 Textbook Contributors