What Is Inference Engineering?

This opens a new series — Deep Learning Inference Engineering. Lesson 1, Part 1 asks the question the whole series exists to answer in increasing depth: what does it actually mean to run a trained model, and why is doing that well an engineering discipline in its own right rather than an afterthought once training is done?

1. Training and Inference Are Different Problems

1.1 The function view

Strip away the training loop, the loss curves, and the dataset pipeline, and a trained machine-learning model is a single mathematical object: a function.

y = f(x; θ)

Here x is the input, θ is the model's learned parameters (weights, biases, batch-norm statistics — every number the model needed to learn), and y is the output. The semicolon in f(x; θ) is doing real work: it separates the variable the function is evaluated on (x) from the variable that defines which function this is (θ). Change x and you get a different prediction from the same model. Change θ and you get a different model entirely.

Training and inference are two completely different operations on this one object, and confusing them is the single most common source of muddled thinking about "AI performance" in casual conversation.

Training searches for the value of θ that minimizes a loss function L over a dataset of labeled examples:

θ* = argmin over θ of  L(f(x; θ), y)

In plain language: try many settings of θ, measure how wrong the model's predictions are against known correct answers, and adjust θ — via gradient descent, using gradients computed by backpropagation — to make the predictions less wrong. Training is a search problem over a parameter space that, for a modern model, can have anywhere from millions to hundreds of billions of dimensions.

Inference takes the θ* that training already found, treats it as a constant, and simply evaluates the function once for a new input:

y = f(x; θ*)

That's it. No search. No loss function evaluated against a ground-truth label (the label is exactly what you're trying to produce). No adjustment of θ*. Inference is function evaluation, full stop — and everything this series covers is about how to make that one evaluation fast, cheap, and reliable at scale.

1.2 Worked example: an image classifier

Make this concrete with a standard image classifier — something like a ResNet-style convolutional network that maps a 224×224 RGB image to one of 1,000 class labels.

What θ actually is. For this model, θ is the complete set of learned numbers: every convolution filter's weights (tens of thousands of 3×3 or 1×1 kernels, each a small grid of floating-point numbers), every batch-normalization layer's learned scale and shift, and the final fully-connected layer's weight matrix mapping the last feature vector to 1,000 class logits. For a ResNet-50, that's about 25 million individual float32 numbers — roughly 100 MB of parameters sitting in memory, fixed, never touched by inference.

What a forward pass computes. Given one 224×224×3 image, the forward pass runs it through a fixed sequence of operations: a stem convolution, a stack of residual blocks (each a small chain of convolutions, batch norms, and ReLU activations, plus a skip connection), a global average pool, and a final matrix multiply against the classifier weights to produce 1,000 logits, followed by a softmax to turn those logits into probabilities. Every one of those operations is deterministic given x and θ* — the same image in always produces the same logits out. This entire sequence of operations, wired together, is f(·; θ*) for this model.

What is entirely absent. Training this same model additionally requires: a loss function (typically cross-entropy against the true label), a backward pass that applies the chain rule through every operation in reverse to compute the gradient of the loss with respect to every one of those 25 million parameters, and an optimizer step (SGD, Adam, or similar) that uses those gradients — plus, for adaptive optimizers, per-parameter running statistics — to nudge θ toward θ*. None of that exists at inference time. There is no backward graph, no gradient tensors, no optimizer state, and for a well-designed inference runtime, no label at all.

This absence is not a minor implementation detail — it is the single biggest reason inference and training have such different memory footprints and performance profiles for the same model. Training a model with N parameters using Adam needs, at minimum, four categories of state living in memory simultaneously:

That's already 4× the parameter memory before counting a single byte of activation memory needed to support the backward pass. Inference needs only the parameters and the activations required to evaluate the current forward pass — and because nothing needs to be retained for a backward pass that will never happen, an inference runtime can even discard each layer's intermediate activations as soon as the next layer has consumed them, which is exactly what memory planning (Section 4) is built around.

Training asks "how do we find a good θ?" Inference asks "how do we evaluate f(x; θ*) as efficiently as possible, given that θ* is already fixed?" Every optimization technique this series covers — quantization, kernel fusion, graph compilation, memory planning — is a way of answering the second question, and every one of them would be meaningless applied to the first.

1.3 Why the distinction matters for engineering, not just semantics

Because inference has no backward pass, no gradient computation, and no optimizer state, it opens up an entire category of optimizations that would be unsound or simply undefined during training: converting weights from float32 to int8 (a lossy transformation gradients would not tolerate well without careful handling), fusing multiple operations into a single kernel so intermediate results never touch memory, discarding activations the instant they're consumed, and picking a completely different numerical algorithm for the same mathematical operation because only the forward result needs to match, not its derivative. Training and inference are optimized by largely disjoint toolkits precisely because they are, underneath the shared vocabulary of "neural networks," different computational problems.

2. Defining Inference Engineering

Given that inference is "just" evaluating f(x; θ*), why does an entire engineering discipline exist around it? Because evaluating that function well — under the constraints of real hardware, real users, and real budgets — is far harder than evaluating it correctly. Correctness is table stakes; a for-loop implementing the model's math will produce the right answer. Inference engineering exists for everything past that point.

A working definition:

Inference engineering is the practice of designing and optimizing the full software and hardware path — model representation, compiler, runtime, kernels, memory system, and hardware execution strategy — needed to execute a trained model efficiently under the constraints of a real deployment target.

That definition earns its keep once you write it as an optimization problem instead of a sentence. Inference engineering, formally, is:

minimize    Cost(Latency, Throughput, Memory, Power, Energy)
 
subject to  Accuracy ≥ A_min
            Latency  ≤ L_max
            Memory   ≤ M_max
            Power    ≤ P_max

Read this the way you'd read any constrained optimization: there is an objective (some combined cost across latency, throughput, memory, power, and energy — the exact weighting depends entirely on the deployment) and a set of hard constraints that any acceptable solution must satisfy regardless of how good the objective looks. A solution that violates Accuracy ≥ A_min is not "a fast model with slightly lower accuracy" — it's a rejected solution, because accuracy below the floor makes the model useless for its task no matter how cheap it is to run. Same for a solution that blows the memory budget on a device with a fixed amount of RAM: it doesn't run at all, so its latency number is irrelevant.

This is why inference engineering is not "making neural networks faster." It is a constrained multi-objective optimization problem across the entire computing stack — and a solution that improves one term of the objective while silently violating a constraint is not an optimization, it's a bug.

The rest of this lesson unpacks both halves of that formulation: what the constraints actually are (Section 3), and what "the entire computing stack" concretely consists of (Section 4).

3. The Constraint Space

3.1 The constraints, one at a time

The optimization problem above compresses a dozen distinct, sometimes competing, real-world constraints into five symbols. It's worth unpacking each on its own before looking at how they interact.

Two of these deserve a second look because they are frequently conflated: power and energy. Power is instantaneous — watts, right now — and it's what determines whether a chip throttles or a battery-powered device gets uncomfortably warm in your hand. Energy is power integrated over time — joules for the whole inference — and it's what determines how many inferences a battery charge can sustain. A model that runs in 10 ms at 2 W and a model that runs in 40 ms at 0.5 W both cost roughly 20 mJ per inference — same energy, very different latency and peak power profile. Depending on whether you're optimizing for "doesn't overheat" or "lasts all day on one charge," these two constraints pull toward different designs.

3.2 A worked tension: latency versus throughput via batch size

Constraints in this list are not independent knobs — improving one routinely costs another, and the cleanest illustration is batch size.

Run inference one input at a time (batch size 1), and each request is processed the moment it arrives: minimal queuing delay, minimal latency. But most inference hardware — GPUs and NPUs especially — is built around large arrays of parallel compute units (matrix-multiply engines, SIMD lanes) that are only fully occupied when there's enough independent work to spread across them. A batch of one input leaves most of that hardware idle waiting on memory, so the hardware utilization constraint suffers and the throughput achieved (inferences per second, amortized over many requests) is low.

Now batch many inputs together — 32, 64, 128 at once — and the same weight values get reused across every input in the batch before being evicted from cache, the parallel compute units stay busy, and throughput climbs substantially. But an individual request now has to wait for the batch to fill before processing even starts, and then wait for the entire batch to finish before its own result comes back. Latency for that one request goes up, even though the hardware is being used far more efficiently.

You cannot maximize latency and throughput simultaneously with a single batch-size choice on fixed hardware. Smaller batches favor latency; larger batches favor throughput. This tension — and the scheduling strategies (dynamic batching, continuous batching) built to manage it — is covered in full in a later post in this series. For now, the point is narrower: the tension is real, it is structural, not a bug to be engineered away, and any serving system has to choose a point on that tradeoff curve deliberately rather than by accident.

The same pattern — one lever, two constraints pulling in opposite directions — recurs throughout inference engineering: precision reduction trades accuracy for memory and energy; kernel fusion trades flexibility for latency; aggressive prefetching trades memory footprint for reduced stalls. Recognizing "this is a tradeoff, not a free win" is often the first useful diagnostic step when someone claims an optimization is strictly better.

4. The Inference Stack

Section 2's definition mentioned "the entire computing stack." Here is what's actually in it, and — more importantly — why it's organized into these particular layers rather than being one monolithic piece of code.

Trained Model (architecture + weights θ*)
       │
       ▼
Model Representation        (ONNX, TorchScript, SavedModel, GGUF, ...)
       │
       ▼
Graph Representation          (a DAG: nodes are ops, edges are tensors)
       │
       ▼
Compiler / IR                   (MLIR, XLA HLO, TVM Relay, TensorRT builder, ...)
       │
       ▼
Graph & Kernel Optimization        (fusion, layout choice, constant folding, kernel selection)
       │
       ▼
Memory Planning                       (buffer allocation, lifetime analysis, arena reuse)
       │
       ▼
Runtime / Scheduler                     (session init, execution providers, op dispatch)
       │
       ▼
Hardware Execution                        (CPU SIMD units, GPU SMs, NPU MAC arrays)

4.1 Model representation

The first thing inference needs is a serialized, framework-independent description of the model: its architecture (which operations, wired together how) and its weights (θ*, saved to disk). Formats like ONNX, TorchScript, TensorFlow's SavedModel, or GGUF exist because a model trained in one framework (PyTorch, JAX, TensorFlow) needs to be consumable by a runtime that has no dependency on that training framework at all. This layer is separate from everything below it because it is the interchange boundary — the point where "how the model was trained" stops mattering and "how the model will be executed" begins. Without it, every inference runtime would need to embed an entire training framework just to load a model, which is both wasteful and a maintenance nightmare (training frameworks change fast; a serialized model format should not).

4.2 Graph representation

Once loaded, the model becomes a graph: a directed acyclic graph (DAG) where nodes are operations (convolution, matrix multiply, add, ReLU) and edges are the tensors flowing between them. This is a deliberately more abstract representation than the serialized file format — it exposes the model's structure in a form that's easy for software to analyze and rewrite. This layer exists separately because graph-level reasoning (which nodes can be fused, which are dead ends whose output is never used, which operations commute) is a distinct kind of analysis from either "how is this stored on disk" (Section 4.1) or "how does this one operation execute on this one chip" (Section 4.5) — conflating them would make each concern harder to reason about in isolation.

4.3 Compiler / IR

A compiler translates the graph into one or more intermediate representations (IRs) — MLIR dialects, XLA's HLO, TVM's Relay/TIR — that are progressively lower-level and closer to what specific hardware can execute. This mirrors exactly why traditional compilers (LLVM, GCC) use IRs instead of translating source code directly to machine code: an IR is a stable place to apply target-independent optimizations once, rather than re-implementing the same rewrite for every backend, and it's a stable interface new hardware backends can target without touching the frontend. For deep learning specifically, this layer is where a single ONNX graph can ultimately be compiled to run on an x86 CPU, an Nvidia GPU, or an edge NPU, from the same starting point.

4.4 Graph and kernel optimization

This layer rewrites the graph and chooses concrete implementations. Graph-level rewrites include operator fusion (folding a convolution, a batch-norm, and a ReLU into a single kernel launch so intermediate results never round-trip through memory), constant folding (pre-computing anything that doesn't depend on the runtime input), and layout transformation (choosing NCHW versus NHWC tensor layout to match what a target's kernels actually want). Kernel selection picks, for each remaining operation, which concrete implementation to run — a naive direct convolution, an im2col-plus-GEMM convolution, or a Winograd-transformed convolution (Section 5.3) can all compute the same mathematical result, and picking correctly for the specific tensor shapes and target hardware is exactly the job of this layer. This is a separate layer from graph representation because it needs hardware-specific cost information (which kernels exist for this backend, roughly how fast each one is) that the earlier, target-independent graph layer deliberately doesn't have.

A small before/after makes fusion concrete. Before optimization, a convolution followed by batch normalization followed by a ReLU is three separate ops, each reading its input from memory and writing its output back before the next op starts:

Before fusion (3 kernel launches, 2 round-trips to memory for intermediates)
    x  → [Conv]  → t1 → [BatchNorm] → t2 → [ReLU] → y
                    (t1, t2 both written to and re-read from memory)
 
After fusion (1 kernel launch, 0 round-trips for intermediates)
    x  → [Conv + BatchNorm + ReLU, fused]  → y
                    (t1, t2 exist only in registers, never touch memory)

Mathematically the two versions compute the identical result. Operationally, the fused version eliminates two full tensor writes and two full tensor reads — exactly the kind of memory traffic Section 5 shows can dominate wall-clock time even when it changes the FLOP count by nothing at all.

4.5 Memory planning

Before execution, the runtime decides where every tensor lives in memory and for how long. Because inference has no backward pass to support (Section 1.3), a tensor's memory can be reclaimed the instant nothing downstream still needs it — this is a lifetime-analysis problem structurally identical to register allocation in a traditional compiler. A good memory planner builds a small number of reusable buffer "arenas" and packs tensors into them based on their computed lifetimes, so peak memory usage reflects the model's actual live working set rather than the sum of every intermediate tensor ever produced. This is a distinct layer from kernel optimization because it operates on the whole graph's tensor lifetimes at once, after kernel selection has fixed exactly what buffers will exist and in what order — trying to do memory planning and kernel selection simultaneously would make both problems far harder to reason about.

4.6 Runtime and scheduler

The runtime is what actually exists at execution time: it loads the compiled graph and its planned buffers, initializes an inference session, dispatches each operation to the hardware backend responsible for it (ONNX Runtime calls this an execution provider — CPU, CUDA, TensorRT, and others can all be registered and mixed within a single graph), and manages the control flow of running the graph, including any threading or asynchronous scheduling. This layer is separate from compilation because compilation happens once (potentially offline, ahead of deployment) while the runtime executes potentially millions of times against that one compiled artifact — keeping them separate is what makes "compile once, run many times cheaply" possible at all, and it's exactly the online-versus-offline optimization split that ONNX Runtime's own documentation describes: graph optimizations can be applied once and serialized to disk, so every later inference session skips re-doing that work.

4.7 Hardware execution

Finally, the actual arithmetic happens on real silicon — CPU vector units (AVX, NEON), GPU streaming multiprocessors, or an NPU's array of multiply-accumulate (MAC) units, each with its own memory hierarchy, parallelism model, and quirks. Every layer above exists to feed this layer well: to give it large enough, well-shaped chunks of work that it can run near its theoretical peak instead of stalling on data it doesn't have yet. Which is exactly the failure mode the rest of this lesson is about.

Each layer in this stack exists because it answers a genuinely different question — "how is the model stored," "what does the computation graph look like," "how do we lower this to a specific chip," "which concrete kernel wins," "where does each tensor live and for how long," "who dispatches work at runtime," "how does the hardware itself execute one operation" — and collapsing any two of those questions into one undifferentiated blob makes each one harder to answer well.

5. Computation Is Not the Whole Problem

5.1 The naive assumption

It's tempting to estimate a model's inference cost by counting its FLOPs (floating-point operations) and comparing that number against a chip's peak FLOP/s rating. A 4 GFLOP model on a chip rated at 4 TFLOP/s "should" take about 1 ms. In practice, real inference latency for that same model can easily be 10 to 50 times worse than that estimate — not because the FLOP count was wrong, but because FLOPs measure only the arithmetic, and arithmetic is frequently not what the hardware spends its time waiting on.

Performance ≠ FLOPs alone.

The missing variable is data movement: how many bytes have to travel between DRAM and the compute units to support those FLOPs, and how many of those bytes are the same bytes being re-fetched over and over because an implementation failed to reuse data already sitting in fast on-chip memory. Two implementations of the exact same mathematical operation, doing the exact same number of FLOPs, can have wildly different memory traffic — and the one that moves more bytes loses, regardless of how clean its arithmetic looks on paper.

5.2 Worked example: two convolutions, identical FLOPs, very different memory traffic

Take a single mid-network convolution layer: a 3×3 kernel, stride 1, 128 input channels, 128 output channels, on a 56×56 spatial output (a realistic shape from the middle stages of a ResNet-style network), computed in float32.

Step 1 — the FLOP count, which is fixed by the math and does not depend on implementation.

MACs  = K² × Cin × Cout × Hout × Wout
      = 9 × 128 × 128 × 56 × 56
      = 9 × 16,384 × 3,136
      = 462,422,016 MACs
 
FLOPs = 2 × MACs  (each multiply-accumulate is one multiply + one add)
      ≈ 924.8 million FLOPs  (≈0.92 GFLOPs)

Both implementations below perform exactly this many floating-point operations. Neither skips work or does extra arithmetic. If FLOP count alone predicted performance, they would run in identical time.

Step 2 — implementation A: a naive loop order.

Suppose the loops are ordered output-pixel-outer: for every one of the 3,136 output pixels, the innermost loops sweep across all 128×128×9 weights to accumulate that pixel's result, and the loop structure never explicitly keeps the weight tensor resident in on-chip cache across pixels. The weight tensor is:

WeightBytes = 3 × 3 × Cin × Cout × 4 bytes
            = 9 × 128 × 128 × 4
            = 589,824 bytes  (576 KB)

If that 576 KB tensor gets evicted and re-fetched from DRAM once per output pixel — a realistic outcome for a naive loop order on hardware where 576 KB doesn't comfortably coexist in cache alongside the input tile and other working data — total weight traffic for this one layer is:

NaiveWeightTraffic = Hout × Wout × WeightBytes
                    = 3,136 × 589,824 bytes
                    ≈ 1.85 GB

Step 3 — implementation B: a tiled loop order.

Now block the output into 8×8 tiles (56 divides evenly into seven tiles of eight), and restructure the loops so the entire 576 KB weight tensor is loaded once per tile and reused across all 64 output pixels in that tile before being evicted — the same weight-stationary idea used by any well-tiled matmul or convolution kernel.

NumTiles = (56 / 8) × (56 / 8) = 7 × 7 = 49
 
TiledWeightTraffic = NumTiles × WeightBytes
                    = 49 × 589,824 bytes
                    ≈ 28.9 MB

Step 4 — convert to wall-clock time using a representative edge-NPU memory bandwidth (≈50 GB/s) and compute throughput (≈1 TFLOP/s).

Compute time (both implementations, identical FLOPs):
    924.8M FLOPs / 1e12 FLOP/s ≈ 0.93 ms
 
Naive:  weight-traffic time  ≈ 1.85 GB  / 50 GB/s ≈ 37.0 ms
Tiled:  weight-traffic time  ≈ 28.9 MB  / 50 GB/s ≈ 0.58 ms

Implementation A performs the identical 924.8 million FLOPs as implementation B, on the identical hardware, and is roughly 40 times slower — dominated entirely by DRAM traffic that the tiled version simply never generates. On the naive implementation, an engineer staring only at the FLOP count would predict sub-millisecond execution and be off by more than an order of magnitude. On the tiled implementation, memory traffic has been reduced to the point where the layer is close to compute-bound — actual runtime tracks the FLOP-based estimate reasonably well, which is precisely why "count the FLOPs" only works once memory traffic has already been engineered down to a non-issue.

The naive implementation loses not because it does more arithmetic — it does exactly the same arithmetic — but because it fails to reuse data that was already available on-chip. Identical FLOPs, a ~64× difference in bytes moved (matching the tile size, exactly as reuse theory predicts), and a ~40× difference in wall-clock time.

5.3 When fewer FLOPs still lose

The reverse failure mode is just as instructive, and just as common in real compiler and runtime engineering: an algorithm with a genuinely lower FLOP count can still lose to one with a higher FLOP count, for the same underlying reason.

Winograd-based convolution (the fast-convolution algorithms popularized by Lavin and Gray for exactly this use case) reformulates small-kernel convolutions — 3×3 is the canonical case — to reduce the number of multiplications by roughly 2.25× relative to direct convolution, using transform matrices applied to input tiles, filters, and outputs. On paper this looks like a strictly better deal: fewer multiplies for the identical mathematical result. In practice, real compilers and inference runtimes (TVM and TensorRT among them) apply Winograd selectively rather than universally, because the transform steps introduce their own memory traffic — extra intermediate buffers for the transformed input, transformed weights, and transformed output tiles — and that overhead does not shrink proportionally when channel counts are small, as they often are in a network's earliest layers. For those shapes, a well-tiled direct or GEMM-based convolution, despite its higher nominal FLOP count, can match or beat Winograd once actual memory traffic and per-tile overhead are accounted for.

The lesson is the same one from Section 5.2, viewed from the opposite direction: the FLOP count is a real number and a useful first estimate, but it is a description of the arithmetic alone. Whether an implementation is fast is a statement about the whole system — arithmetic, memory traffic, cache behavior, and overhead together — which is exactly why inference engineering treats hardware utilization, not FLOP count, as the constraint that actually predicts wall-clock performance (Section 3.1's last row, and the subject of the roofline model in the next post).

Further Reading

• Williams, S., Waterman, A., & Patterson, D., "Roofline: An Insightful Visual Performance Model for Multicore Architectures", Communications of the ACM, Vol. 52, No. 4 (2009) — the original formalization of arithmetic intensity and the compute-bound/memory-bound framing this lesson leans on in Section 5, developed in full in Lesson 1, Part 2.
• Sze, V., Chen, Y.-H., Yang, T.-J., & Emer, J. S., Efficient Processing of Deep Neural Networks (Synthesis Lectures on Computer Architecture) — the standard systems-level treatment of the full inference stack, from algorithms down to hardware datapaths and dataflows; see also their earlier survey, "Efficient Processing of Deep Neural Networks: A Tutorial and Survey".
• ONNX Runtime, "Execution Providers" — the real, production documentation for how a single compiled graph dispatches different subgraphs to different hardware backends, referenced in Section 4.6.
• ONNX Runtime, "Graph Optimizations in ONNX Runtime" — documents the provider-independent versus provider-specific optimization passes and the online/offline optimization split referenced in Section 4.6.
• Cloudflare, "AI Inference vs. Training: What Is AI Inference?" and GeeksforGeeks, "What is AI Inference in Machine Learning?" — accessible framings of the training/inference split covered formally in Section 1.

Section 5 ended by naming the real diagnostic tool for separating compute-bound work from memory-bound work: arithmetic intensity, plotted against a hardware's peak compute and peak bandwidth on a single chart. Lesson 1, Part 2 — Compute-Bound, Memory-Bound, and the Roofline Model — builds that chart from first principles and uses it to explain, precisely, why the naive convolution in Section 5.2 lost.