The full stack of running a trained model fast: compute vs memory-bound analysis, quantization, kernel engineering, memory planning, runtime scheduling, transformer inference, and hardware co-design.
Lesson 1, Part 1 of a new series on deep learning inference engineering: the training/inference distinction, a formal definition of inference engineering as a constrained optimization problem, a full walk through the inference stack, and why raw FLOP counts don't predict real-world performance.
A full derivation of arithmetic intensity and the Roofline model, worked through real GEMM, convolution, and operator-fusion examples, to answer the one question that determines every subsequent optimization decision: is this kernel starved for data, or starved for ALUs?
How compilers pick physical tensor layouts, block computation to fit on-chip memory, fuse operators to avoid materializing intermediates, and fold or prune away work that never needed to happen at runtime.
A from-scratch derivation of affine quantization's scale and zero-point formulas, worked numeric examples in int8, and why per-channel quantization is often the difference between a working model and a broken one.
How an int8 convolution actually computes under the hood — int32 accumulation, requantization, and the calibration and training techniques (min/max, percentile, entropy, and QAT) that decide whether the resulting int8 model is still accurate.
A tour of the nine largely-independent levers that separate a mathematically-correct kernel from a fast one — loop order, blocking, cache and register reuse, prefetching, alignment, threading, and specialized ISA instructions — worked through the classic ikj-vs-ijk matmul flip and a from-scratch 4x4 register-blocked micro-kernel.
A precise, arithmetic-backed tour of SIMD vectorization, a from-scratch derivation of why GEMM is the central kernel of deep learning and how BLIS-style blocking makes it fast, and a fully worked numeric example showing exactly how im2col turns convolution into matrix multiplication.
How inference runtimes decide, ahead of time, exactly where every intermediate tensor will live — a worked graph-coloring example in static memory planning, plus zero-copy techniques like in-place ops, view-only reshapes, and memory-mapped weights.
How an inference runtime decides which device runs each operator, why crossing between CPU and NPU carries a real synchronization cost, and how double-buffered DMA can hide nearly half of a pipeline's total time — with the arithmetic worked out in full.
A precise, first-principles treatment of latency vs throughput in ML serving, with a worked roofline-style derivation of why batching restores arithmetic intensity, a queueing-delay derivation of what batching costs, and the adaptive-batching-with-timeout design real serving systems use to balance the two.
A first-principles derivation of the KV cache — why it exists, what it costs in memory, and how it splits LLM inference into a compute-bound prefill phase and a memory-bound decode phase — closing Lesson 6 by extending Lesson 1's roofline results to real transformer serving.
Why no single IR can be both close to a framework's semantics and close to hardware, walked through MLIR's real dialect stack, and how that multi-level structure is exactly what makes layout transformation and kernel selection tractable compiler passes instead of unsolvable one-shot decisions.
A first-principles tour of pruning, knowledge distillation, and sparse inference: why zeroing out weights rarely makes a model faster on its own, how a teacher's soft labels teach a student more than ground-truth labels ever could, and why NVIDIA's 2:4 structured sparsity is the one sparsity pattern real hardware can actually exploit.
A precise treatment of why intuition about bottlenecks is usually wrong, why wall-clock time and hardware performance counters answer different questions, and why percentile-based benchmarking is the only honest way to report latency for a production inference system.
Amdahl's Law turns a profile into a spending plan for where to optimize, a wearable's 100 Hz sensor-fusion loop turns that plan into a hard 10 ms deadline, and for battery-powered inference the real constraint often isn't time at all — it's joules, where moving a byte from DRAM can cost two orders of magnitude more energy than computing on it.
Why microcontroller-class inference is a different discipline from anything upstream of it — real numbers on the RAM/flash gap versus phones and servers, why int8 quantization stops being optional once the FPU disappears, the FLOP arithmetic behind depthwise-separable convolutions, and why static memory planning becomes the only way the system boots at all.
How the systolic array turns a matmul into a timed wave of data flowing through a MAC grid instead of a stream of individual instructions, why an NPU's on-chip SRAM is a scratchpad the compiler owns rather than a cache the hardware manages, and how weight-, output-, and activation-stationary dataflows force tiling, loop order, and operator placement to become hardware-specific compiler decisions — closing with all six of this lesson's optimizations stacked into one 14x speedup.
One model gets six optimizations stacked on top of each other for a roughly 14x illustrative speedup, then the post works backward through the eight mistakes that most often erase gains like that, the five-term equation that explains where every millisecond actually went, and the order in which a practitioner should attack the problem — with real cross-references back to Lessons 1 through 10.
The finale of the Deep Learning Inference Engineering series: how a C++ and MLIR compiler background maps directly onto the whole inference stack, a five-dimension mental model that ties every lesson together, a practical model-diagnosis checklist, a seven-level learning path, and six hands-on projects to turn all of it into real skill.