Distributed ML Notes
Engineering notes on distributed ML systems — written from an infrastructure perspective. The focus is on the system design problems underneath the ML abstractions: data movement at scale, distributed compute coordination, memory hierarchies, and the operational failure modes that textbooks skip.
Content keep updating...
live site at : https://lucyge2022.github.io/Distributed-ML-Notes/index.html
Contents
Concepts
Background knowledge that cuts across all chapters.
- GPU, VRAM, PCIe & NVLink — hardware mental model: why GPUs, what VRAM is, PCIe bottleneck, NVLink
- PyTorch Memory Management — how PyTorch allocates and caches CUDA memory
Chapter 2 — Data Ingestion Patterns
How raw data gets from disk into a training loop.
- Dataset — dataset composition, IDX binary format, tensor representation, data ingestion flow
- Batching, Sharding & Caching — Ray+Parquet vs WebDataset vs MosaicML MDS: formats, sharding strategies, shuffle, remote streaming
Chapter 3 — Distributed Training Patterns
How training is parallelized across multiple machines.
- Parameter Server Pattern — PS + workers architecture, gradient flow, staleness, fault tolerance
- Worker-Only (AllReduce) Pattern — Ring AllReduce, PyTorch DDP, fault tolerance, failure analysis
Chapter 4 — Feature Store
How features are computed, stored, and served consistently between training and inference.
- Feature Store — offline vs online paths, train-serve skew, Chronon consistency measurement; illustrated with a user–restaurant recommendation system
Supplemental Code
Runnable toy programs that accompany these notes: distributed-ml-examples
| Example | What it covers |
|---|---|
| ddp-testrun | PyTorch DDP training with Ring AllReduce on MNIST |
Built with
mdBook — the src/SUMMARY.md defines the book structure.
mdbook serve # local preview at http://localhost:3000
mdbook build # build static site to book/