efficient-dl-systems

Efficient Deep Learning Systems

This repository contains materials for the Efficient Deep Learning Systems course, taught at the Faculty of Computer Science of HSE University and Yandex School of Data Analysis.

This branch corresponds to the 2026 iteration of the course. If you want to see full materials of past years, see the "Past versions" section.

Syllabus

• Week 1: Introduction
  • Lecture: Course overview and logistics. Core concepts of the GPU architecture and the CUDA API.
  • Seminar: CUDA operations in PyTorch. Introduction to benchmarking.
• Week 2: General training optimizations, profiling deep learning code
  • Lecture: Measuring performance of GPU-accelerated software. Mixed-precision training. Data storage and loading optimizations. Tools for profiling deep learning workloads.
  • Seminar: Automatic Mixed Precision in PyTorch. Basics of profiling with py-spy, PyTorch Profiler, Memory Snapshot and Nsight Systems. Dynamic padding for sequence data and JPEG decoding benchmarks.
• Week 3: Data-parallel training and All-Reduce
  • Lecture: Introduction to distributed training. Data-parallel training of neural networks. All-Reduce and its efficient implementations.
  • Seminar: Introduction to PyTorch Distributed. Data-parallel training primitives.
• Week 4: Methods for training large models
  • Lecture: Tensor, pipeline, sequence parallelism. Gradient checkpointing, offloading.
  • Seminar: Gradient checkpointing and tensor parallelism in practice.
• Week 5: Sharded data-parallel training, distributed training optimizations
  • Lecture: FSDP training and its optimizations.
  • Seminar: In-depth overview of PyTorch Device Mesh, DTensor, FSDP2, and Distributed Checkpoint.
• Week 6: Deep learning performance from first principles.
  • Lecture: Arithmetic of Deep Learning
  • Seminar: Kernel fusion, torch.compile, GPU memory hierarchy and Liger kernels
• Week 7: Basics of web service deployment.
  • Lecture/Seminar: Web service deployment and production-ready systems basics.
• Week 8: Systems optimizations for inference
  • Lecture: Inference speed metrics. KV caching, batch inference, continuous batching. FlashAttention with its modifications and PagedAttention. Overview of popular LLM serving frameworks.
  • Seminar: KV cache utilization, continuous batching for high-throughput serving, and performance measurements of the prefill and decode stages.
• Week 9: Algorithmic optimizations for inference.
  • Lecture: LLM inference use cases and metrics. Architecture optimizations, quantization, speculative decoding, KV cache compression, knowledge distillation.
  • Seminar: Proof of distribution preservation in speculative decoding. Matrix multiplication in Triton for different scenarios. Using nsys profile.
• Week 10: Guest lecture

Grading

There will be several home assignments (spread over multiple weeks) on the following topics:

• Training pipelines and code profiling
• Distributed and memory-efficient training
• Deploying and optimizing models for production

The final grade is a weighted sum of per-assignment grades. Please refer to the course page of your institution for details.

Staff

• Max Ryabinin
• Just Heuristic
• Yaroslav Zolotarev
• Ruslan Khaidurov
• Maksim Abraham
• Sergei Vorobyov
• Gregory Leleytner
• Antony Frolov
• Mikhail Khrushchev
• Anton Chigin
• Kamil Izmailov
• Roman Gorb
• Mikhail Seleznev

Past versions

• 2025
• 2024
• 2023
• 2022
• 2021