AI systems Performance Engineering Manual

Most modern graphics hardware is capable of extraordinary throughput, and most deep learning workloads never get close to using it. A training job running at a fifth of what the hardware can deliver is not an edge case, it is the default starting point for most teams, and it carries a direct cost in training time, serving capacity, and cloud spend.

The usual fixes, random code changes, a hardware upgrade, or a trick borrowed from a forum post, rarely close that gap, because the real bottleneck is almost never visible in the Python code itself. It lives deeper: in how threads are scheduled, how memory moves, how many small operations launch one after another, and how multiple processors coordinate once a job spans more than one device.

This book builds one complete, consistent model of that picture and pairs it with a disciplined measurement process, carrying the same reasoning from a single processor to a large distributed cluster and a live production serving system.

What this book helps you do:

• Classify any workload as limited by computation or by data movement, using a simple hardware-based model.
• Use a profiling toolchain to turn "this feels slow" into a precise, evidence-based diagnosis.
• Apply reduced-precision numeric formats correctly, including techniques that keep training stable.
• Manage memory directly, including allocator behavior, fragmentation, and checkpointing tradeoffs.
• Configure a multi-device training strategy, data-parallel, model-parallel, or a pipelined mix, matched to your cluster.
• Design a serving system around realistic batching, caching, and latency targets instead of trial and error.
• Work through fully solved case studies and triage checklists that turn the material into a repeatable skill.

Key topics include processor architecture, memory hierarchy and bandwidth, kernel profiling and trace analysis, operator fusion and graph compilation, custom kernel programming, mixed precision and quantization, memory optimization, multi-device communication, distributed training strategies across several parallelism approaches, and production inference-serving performance.

Written for engineers who already build and train models in a modern Python-based deep learning framework and are ready to ask why a workload runs slower than it should: machine learning engineers, infrastructure and platform engineers, applied researchers, and technical leads responsible for the speed or cost of a training or serving system, from one workstation to a large cluster.

Get your copy and start closing the gap between what your hardware can do and what your workloads currently achieve, with one dependable guide that carries the same reasoning from a single processor to an entire production system.

Trademarked Disclaimer
This is an independent publishers book, it's not affiliated with, endorsed with, sponsored by or endorsed by any organizations.