Tensor Layouts, Tiling, and Graph-Level Optimization

Lesson 1 established the memory wall and the roofline model: on real hardware, moving data usually costs more than computing on it, and arithmetic intensity determines whether a kernel is memory-bound or compute-bound. Lesson 2 turns that theory into the actual levers an inference compiler pulls before a single MAC executes — how tensors sit in memory, how big a chunk of them touches the compute array at once, which operators get welded together, and which parts of the graph the compiler can simply refuse to run.

1. Tensors Are Abstract, Memory Is Not

A tensor with shape (N, C, H, W) is a mathematical object — four indices, one value per coordinate. But a physical memory system has no concept of four dimensions. DRAM and SRAM are linear: one address space, one byte offset per location. Every multi-dimensional tensor has to be flattened into that one-dimensional space, and the order in which you flatten it is a real, consequential design decision — not a formatting detail.

This is the same idea Lesson 1 built the entire cache-address derivation around: hardware doesn't reason about "arrays," it reasons about addresses and strides. A layout choice is really a choice about which logical neighbors end up as physical neighbors — and physical neighbors are exactly what a cache line, a burst DRAM read, or a vector load unit reward.

Two layouts dominate CNN inference:

1.1 NCHW — channel-major

Here the fastest-varying (contiguous, stride-1) dimension is width, then height, then channel, then batch. Walking memory linearly means: sweep across one entire channel's spatial plane before moving to the next channel.

1.2 NHWC — channel-minor

Here the fastest-varying dimension is channel. Walking memory linearly means: sweep across all channels at a single spatial position before moving to the next pixel.

Both describe the exact same logical tensor and the exact same values. What differs is purely which elements sit next to each other in physical memory — and that difference changes everything about how a convolution kernel accesses them.

2. A Worked Example: The Same Tensor, Two Stride Patterns

Take a small, concrete tensor: shape (N=2, C=3, H=4, W=4) — 2 images, 3 channels, a 4×4 spatial plane, float32 elements (4 bytes each). Total element count: 2 × 3 × 4 × 4 = 96 elements, 384 bytes.

2.1 NCHW strides

For a row-major (C-contiguous) layout with dimension order (N, C, H, W), the stride for each dimension — how many elements you skip in linear memory to advance one step along that dimension — is the product of the sizes of all dimensions to its right:

stride_W = 1
stride_H = W            = 4
stride_C = H × W        = 16
stride_N = C × H × W    = 48

So element (n, c, h, w) lives at linear offset n·48 + c·16 + h·4 + w. Walking w from 0 to 3 moves 1 element at a time — perfectly contiguous. Walking c from 0 to 2 (fixed n, h, w) jumps 16 elements — a full spatial plane — at a time.

NCHW linear memory (batch 0 only), element index = c*16 + h*4 + w
 
Channel 0 (offsets 0-15):    [ 0  1  2  3 | 4  5  6  7 | 8  9 10 11 | 12 13 14 15 ]
                               h=0 row      h=1 row      h=2 row      h=3 row
Channel 1 (offsets 16-31):   [16 17 18 19 | 20 21 22 23 | ... ]
Channel 2 (offsets 32-47):   [32 33 34 35 | ... ]

A 2D convolution needs, for one output pixel, the same spatial window across all input channels simultaneously (a MAC per channel, per kernel tap, accumulated together). Under NCHW, those per-channel values for a fixed (h, w) are 16 elements apart — three separate, widely-spaced reads for a 3-channel tensor, one per channel plane.

2.2 NHWC strides

Same tensor, dimension order (N, H, W, C):

stride_C = 1
stride_W = C             = 3
stride_H = W × C         = 12
stride_N = H × W × C     = 48

Element (n, h, w, c) lives at offset n·48 + h·12 + w·3 + c.

NHWC linear memory (batch 0 only), element index = h*12 + w*3 + c
 
Pixel (0,0) (offsets 0-2):   [ c0  c1  c2 ]
Pixel (0,1) (offsets 3-5):   [ c0  c1  c2 ]
Pixel (0,2) (offsets 6-8):   [ c0  c1  c2 ]
Pixel (0,3) (offsets 9-11):  [ c0  c1  c2 ]
Pixel (1,0) (offsets 12-14): [ c0  c1  c2 ]
...

Now the three channel values for a single pixel are adjacent — offsets w·3, w·3+1, w·3+2 — a single contiguous 12-byte read for float32, and it lands inside one cache line every time regardless of channel count (up to the line size).

2.3 Why this maps directly onto real hardware

A convolution's innermost reduction is a dot product across the channel dimension (for a 1×1 kernel tap) or across (kh, kw, c) jointly for a full kernel window. Two very different hardware philosophies want two different layouts for that reduction:

NHWC favors channel-vectorized hardware. Mobile NPUs, DSPs (Hexagon-style), and many accelerator MAC arrays are built to load a contiguous vector of channel values in one transaction and broadcast a kernel weight across all of them, one instruction, one memory access, C MACs. Because channels are stride-1 in NHWC, a single vector load naturally gathers exactly the values that get reduced together. This is precisely the reduction that shows up in the "channel-last vector load" pattern this reader would recognize from NPU compiler backends targeting SIMD/vector lanes on the channel axis — the physical layout is chosen to match the width of the vector unit, not out of convention.
NCHW favors spatially-vectorized hardware, and it's the historical default on GPUs. cuDNN and classic GPU convolution kernels were designed around NCHW because grouping each channel's full spatial plane together lets a thread block sweep across H×W with simple strided access and keeps im2col-style transformations straightforward. GPU tensor cores also historically expected NCHW-derived tiled layouts for their matrix-multiply formulation of convolution.

Neither layout is universally "correct" — the right choice is whichever one makes the hardware's native reduction pattern a contiguous memory access instead of a strided, cache-line-fragmenting one. This is exactly the same physics as Lesson 1's spatial-locality argument: a cache line rewards you for touching what's next to what you just touched, so the compiler's job is to arrange memory so "next to" in hardware means "next to" in the reduction.

2.4 The cost of getting it wrong — and of converting

If a graph mixes operators that prefer different layouts — a common situation when a framework's default export layout (e.g. NHWC from TensorFlow, NCHW from PyTorch) doesn't match a target backend's preferred layout — someone has to pay for a layout transform: a real memory-bound pass that reads the whole tensor in one stride order and rewrites it in another. This is not free. For a large activation tensor, a layout transpose can cost as much DRAM traffic as the convolution's cheaper layer, and doing it in the wrong place in the graph (e.g. once per layer instead of once at the network boundary) can quietly dominate total inference latency.

This is exactly why graph-level layout optimization exists as its own compiler pass rather than a per-operator decision. TVM's ConvertLayout pass, for example, propagates a single chosen layout through as much of the graph as possible so that only two transforms are needed for the entire network — one converting the input layout in, one converting the output layout back out — instead of one transform sandwiched between every mismatched operator pair. ONNX Runtime's CPU backend similarly applies a layout optimizer (its "NCHWc" transformation) that tiles the channel dimension to match SIMD register width on x86, blending the "vectorize the channel axis" benefit of NHWC-style access into an NCHW-rooted layout.

2.5 Summary comparison

Property	NCHW	NHWC
Fastest-varying (stride-1) dimension	Width	Channel
Per-pixel, all-channel access	Strided (jumps of `H×W`)	Contiguous
Natural fit	GPU tensor-core / cuDNN-style spatial tiling	Vector/SIMD units doing channel-parallel MACs, many mobile NPUs and DSPs
Typical framework default	PyTorch (historically)	TensorFlow / TFLite (historically)
Common compiler response	Native for GPU backends	Preferred entry layout for many edge/NPU backends; compilers insert boundary transforms when mismatched

3. Tiling: Fitting Big Tensors Through a Small Window

Lesson 1's roofline model made the general claim: arithmetic intensity — operations per byte moved — determines whether a kernel is compute-bound or memory-bound, and a naive loop order can make even an O(N³) matmul behave like a memory-bound O(N) elementwise op if it keeps re-fetching the same data from DRAM. Tiling is the concrete mechanism that fixes this: restructure the computation so that once a chunk of data is pulled into fast memory, it gets reused as many times as possible before being evicted.

3.1 The problem in concrete numbers

Suppose an NPU has 512 KB of on-chip SRAM (matching Lesson 1's tiling example scale) and needs to compute C = A × B for two 1024×1024 float32 matrices.

Size of A = 1024 × 1024 × 4 bytes = 4,194,304 bytes ≈ 4 MB
Size of B = same ≈ 4 MB
Size of C = same ≈ 4 MB

None of the three matrices — let alone all three together — fits in 512 KB of SRAM. A naive triple-nested loop computing C[i][j] = sum_k A[i][k] * B[k][j] re-reads a row of A and a column of B from DRAM (or thrashes through cache) on almost every inner-loop step, exactly the pathology Lesson 1 described: high theoretical FLOP count, low realized arithmetic intensity, because reuse never actually happens before eviction.

3.2 Deriving a tile size from the SRAM budget

The fix is to partition each matrix into square tiles of size T × T and restructure the loops so that one tile of A, one tile of B, and one (accumulating) tile of C are resident in SRAM simultaneously, fully consumed, before moving on:

for each tile row  i_t:
  for each tile col j_t:
    load C_tile[i_t][j_t] (accumulator, T×T)
    for each tile k_t:
      load A_tile[i_t][k_t]     (T×T)
      load B_tile[k_t][j_t]     (T×T)
      C_tile += A_tile @ B_tile   ← reused T times before moving on
    store C_tile[i_t][j_t]

Three T×T float32 tiles must fit in the 512 KB SRAM budget simultaneously (with headroom for double-buffering, but ignore that for a first pass):

3 × T² × 4 bytes  ≤  512 × 1024 bytes
T²                ≤  524,288 / 12
T²                ≤  43,690.67
T                 ≤  ~209

Round down to a hardware-friendly power-of-two or SRAM-bank-aligned value: T = 128 is a comfortable, conservative choice that leaves room for double-buffering (loading the next tile while computing on the current one, which roughly doubles the working footprint to 6 tiles instead of 3):

3 × 128² × 4 bytes = 3 × 16,384 × 4 = 196,608 bytes ≈ 192 KB   (single-buffered, fits with room to spare)
6 × 128² × 4 bytes ≈ 393,216 bytes ≈ 384 KB              (double-buffered, still fits under 512 KB)

3.3 Why this raises arithmetic intensity — the arithmetic, explicitly

For one T×T output tile, the compute is a T×T×T matrix multiply:

Operations (this tile) = 2 × T³   (T³ multiply-adds × 2 FLOPs each)
                        = 2 × 128³
                        = 2 × 2,097,152
                        = 4,194,304 FLOPs

The data movement to produce that tile — loading one A tile and one B tile, each once, from DRAM (the whole point of tiling is that each tile, once resident, is reused T times internally without re-touching DRAM):

Bytes moved (this tile) = 2 × T² × 4 bytes
                         = 2 × 128² × 4
                         = 2 × 16,384 × 4
                         = 131,072 bytes

ArithmeticIntensity = Operations / BytesMoved
                     = 4,194,304 / 131,072
                     ≈ 32 FLOP/byte

Compare this to the untiled elementwise-add case from Lesson 1 (≈0.083 FLOP/byte) or to a pathologically untiled matmul that re-fetches every element from DRAM on every reuse (arithmetic intensity collapsing toward that same floor). A ≈32 FLOP/byte tile is nearly 400× denser in useful work per byte moved than the elementwise case, and on most NPU/GPU hardware that is comfortably past the roofline's ridge point — solidly in compute-bound territory, where the SRAM-to-compute-array pipe, not the DRAM bus, is now the bottleneck. Notice this ratio scales with T: doubling the tile dimension roughly doubles arithmetic intensity, which is exactly why the tile-size derivation above matters — it isn't a rounding convenience, it's the knob that directly sets where the kernel lands on the roofline curve.

3.4 The two failure modes, in the same numeric terms

Tile too small (say T = 16): arithmetic intensity drops to 2×16³ / (2×16²×4) = 8192/2048 = 4 FLOP/byte — a 8× worse ratio than T=128, because the fixed cost of loading a tile is amortized over far less reuse. Worse, small tiles mean more tile-boundary overhead (more DMA setup/teardown transactions, more loop iterations, more pipeline bubbles refilling the compute array between tiles) relative to useful work.

Tile too large (say T = 256, ignoring the SRAM budget): 3 × 256² × 4 = 786,432 bytes ≈ 768 KB — this overflows the 512 KB SRAM outright. The hardware or runtime has no choice but to spill: either the tile gets silently split further by a lower software layer (defeating the point of choosing T deliberately), or worse, it thrashes SRAM the same way an oversized working set thrashes an L1 cache in Lesson 1's conflict-miss discussion — data gets evicted before its reuse is exhausted, and effective arithmetic intensity collapses back toward the untiled case despite the code "looking" tiled.

This is precisely the sizing exercise a real NPU compiler backend performs when lowering a large convolution or matmul: read the target's SRAM/scratchpad size from a hardware description, compute the largest tile that keeps all live buffers resident (with margin for double-buffering and any per-tile metadata), and emit loop bounds accordingly — often per-target, since a compiler that emits one universal tile size across very different on-chip memory budgets is leaving performance on the table on every target except the one it was tuned for. TVM's tile scheduling primitive, built from a combination of split (breaking one loop into an outer tile-index loop and an inner intra-tile loop) and reorder (interleaving those loops so tile-local iterations run consecutively), is the standard mechanism compilers use to express exactly this transformation at the IR level.

4. Operator Fusion: Not Writing the Intermediate at All

Tiling optimizes reuse within one operator's execution. Fusion attacks a different, adjacent inefficiency: the round trip between operators.

4.1 The unfused cost

Consider the extremely common CNN pattern Conv → BatchNorm → ReLU. Executed naively as three separate kernels, each one is a complete, independent pass over memory:

Kernel 1 (Conv):
  read input activation  (from DRAM)
  read weights            (from DRAM)
  compute conv output
  WRITE conv output       (to DRAM)         ← intermediate tensor #1, fully materialized
 
Kernel 2 (BatchNorm):
  READ conv output        (from DRAM)       ← re-reads what kernel 1 just wrote
  read BN scale/shift params
  compute normalized output
  WRITE BN output         (to DRAM)         ← intermediate tensor #2, fully materialized
 
Kernel 3 (ReLU):
  READ BN output          (from DRAM)       ← re-reads what kernel 2 just wrote
  compute max(x, 0)
  WRITE final output      (to DRAM)

Every arrow between kernels is a full tensor write followed immediately by a full tensor re-read of the same data, from the same memory tier, milliseconds apart. None of that traffic does any new work — it exists purely because each kernel is a separate compiled unit with its own memory boundary, unaware that its output is about to be immediately consumed by the very next instruction stream.

4.2 The fused version

A compiler that recognizes this pattern can instead emit one kernel that keeps the intermediate values in registers or on-chip SRAM for their entire (very short) lifetime:

Fused kernel (Conv + BatchNorm + ReLU):
  read input activation   (from DRAM)
  read weights            (from DRAM)
  read BN scale/shift params
  for each output element:
    compute conv accumulator            (register/SRAM only)
    apply BN affine transform to it     (register/SRAM only, never touches DRAM)
    apply ReLU to it                    (register/SRAM only, never touches DRAM)
  WRITE final output       (to DRAM)

The two intermediate tensors never exist in DRAM at all. They exist only as values passing through registers or a small SRAM scratch buffer for the few cycles between "conv finished this element" and "ReLU consumed it."

4.3 A rough quantified estimate of the savings

Take a representative activation tensor: 64 channels, 56×56 spatial, float32 — a plausible mid-network feature map size.

Tensor size = 64 × 56 × 56 × 4 bytes = 802,816 bytes ≈ 784 KB

Unfused traffic for the two intermediate tensors (each written once by its producer, read once by its consumer):

Conv output:  write 784 KB + read 784 KB  = 1,568 KB
BN output:    write 784 KB + read 784 KB  = 1,568 KB
Total extra traffic from materializing intermediates ≈ 3,136 KB ≈ 3.06 MB

Fused traffic for the same two boundaries: zero — those bytes simply never leave on-chip storage. So for this one three-op chain, fusion eliminates roughly 3 MB of DRAM traffic per invocation, on top of the traffic every version pays anyway (reading the true input, the weights, and the BN parameters once, and writing the true final output once). If this pattern repeats at every one of, say, 20 conv layers in a network with similarly sized activations, the cumulative avoided DRAM traffic is on the order of tens of megabytes per inference pass — traffic that, per Lesson 1's roofline framing, was pure memory-bound overhead contributing zero useful arithmetic intensity, since BatchNorm and ReLU are themselves nearly free arithmetically (a multiply-add and a max, respectively) compared to the bytes they'd otherwise force through the memory bus.

Fusion also removes kernel launch overhead — each kernel launch on a GPU or NPU carries fixed costs (queue submission, synchronization, pipeline warm-up) independent of tensor size, so collapsing three launches into one removes two of those fixed costs entirely, which matters disproportionately for smaller tensors where the fixed overhead is a larger fraction of total kernel time.

This is documented, production behavior, not a theoretical nicety: ONNX Runtime's basic optimization level explicitly includes semantics-preserving fusions named Conv Add, Conv Mul, Conv BatchNorm, and Relu Clip, and its extended level adds further fusions like Conv Activation, GEMM Activation, and MatMul Add for supported execution providers. TensorRT performs the analogous transformation under the name vertical layer fusion, folding chains of pointwise and convolution-adjacent layers into single optimized kernels for the same reason: fewer launches, fewer intermediate tensors, less memory traffic, better locality.

4.4 Before/after summary

Aspect	Unfused (3 kernels)	Fused (1 kernel)
Kernel launches	3	1
Intermediate tensors materialized to DRAM	2 (conv output, BN output)	0
Extra DRAM traffic from intermediates (784 KB tensor example)	≈3.06 MB (write+read × 2)	0
Arithmetic intensity of BN/ReLU portion	Effectively near-zero (pure memory round trip)	N/A — folded into the conv kernel's intensity
Synchronization / pipeline-refill overhead	Paid 3×	Paid 1×

4.5 Fusion isn't unconditional

Fusion is only free when the fused kernel's working set — inputs, weights, and any needed intermediates — still fits within registers/SRAM for the tile being processed. A compiler that fuses too aggressively (e.g. trying to fuse a huge matmul directly into a large downstream elementwise chain without also tiling the fused result) can create register pressure or SRAM spills that erase the benefit. This is exactly why fusion and tiling are not independent passes in a mature compiler stack — they interact, and scheduling frameworks like TVM's Relay treat "which nodes fuse" and "how the fused result is tiled and scheduled" as a jointly-optimized decision rather than two unrelated steps applied in isolation.

5. Constant Folding: Moving Work from Runtime to Compile Time

Fusion and tiling both optimize work that has to happen at runtime because it genuinely depends on the input. Constant folding attacks a different category entirely: computation that doesn't depend on the input at all, and therefore never needed to happen at inference time in the first place.

5.1 The general principle

If a subgraph's every input traces back only to constants — trained weights, fixed shape parameters, baked-in scalar constants from the original model definition — then its output is itself a constant. There is no reason to recompute that output on every single inference call; it can be computed exactly once, at compile/export time, and stored directly as a constant tensor in the optimized graph.

The runtime graph literally shrinks — the Add(A, B) node and both constant inputs disappear, replaced by a single precomputed constant C. This is the textbook description of constant folding, and ONNX Runtime lists it as a basic-level graph optimization: it statically evaluates any subgraph whose inputs are entirely constant initializers, so that computation never has to be repeated across inference calls.

5.2 The canonical real-world instance: folding BatchNorm into Conv

The single most consequential constant-folding-adjacent optimization in CNN inference is Conv-BatchNorm folding. It's worth walking through precisely, because it's simultaneously a textbook example of "moving work to compile time" and a very real optimization every serious inference stack (ONNX Runtime, TensorRT, TFLite) applies.

At inference time (not training time — the distinction matters, since during training the batch statistics are still being estimated), BatchNorm applies a fixed affine transform to each channel, using statistics that have already converged to running estimates:

BN(x) = γ · (x − μ) / √(σ² + ε) + β

where γ (scale), β (shift), μ (running mean), and σ² (running variance) are all already-fixed constants by the time the model is exported for inference — they were learned or accumulated during training and never change again. ε is a small constant for numerical stability.

A preceding convolution computes, per output channel c:

Conv(x)_c = W_c · x + b_c

Composing BN(Conv(x)) and simplifying algebraically, per output channel c:

BN(Conv(x))_c = γ_c · (W_c · x + b_c − μ_c) / √(σ_c² + ε) + β_c
              = [γ_c / √(σ_c² + ε)] · W_c · x  +  [γ_c · (b_c − μ_c) / √(σ_c² + ε) + β_c]

That is exactly the form of a convolution with new, folded weights and bias:

W_folded_c = W_c · γ_c / √(σ_c² + ε)
b_folded_c = (b_c − μ_c) · γ_c / √(σ_c² + ε) + β_c

Both W_folded and b_folded depend only on already-constant quantities (W, b, γ, β, μ, σ², ε) — none of them depend on the runtime input x. So the entire fold can be computed once, offline, and the exported inference graph can replace the Conv → BatchNorm pair with a single convolution node carrying the folded weights and bias. BatchNorm disappears from the runtime graph entirely — not approximated, not skipped, but algebraically absorbed with zero change to the numerical output (up to floating-point rounding).

This is a strict superset benefit compared to naive operator fusion: fusion (Section 4) would keep BatchNorm as a computation but avoid materializing its intermediate output to memory; folding removes the computation itself, because it was always computable ahead of time. In practice, real compilers apply both ideas together — fold what's foldable first, then fuse whatever compute-dependent chain remains (e.g. the folded Conv with a following ReLU, which does depend on runtime input and so cannot itself be folded).

The savings compound directly with model depth: a ResNet-50-scale network has dozens of Conv-BatchNorm pairs, and folding every one of them removes dozens of full-tensor elementwise passes (a multiply and an add per activation element, across every spatial position and channel) from every single inference call, permanently, for the cost of one offline algebraic simplification per pair.

5.3 Why this is "almost always" a win

Constant folding trades a small, one-time compile-time cost for a runtime cost paid on every inference call, forever. The only situation where it isn't a clear win is if the folded constant is enormous compared to the original operands (rare — folding a Conv+BN pair does not grow the weight tensor's shape) or if the model's constants are expected to change between calls (which would mean they weren't actually constant, and the compiler's constant-detection pass — checking that every input traces to a fixed initializer, not a runtime feed — would correctly refuse to fold them).

6. Dead-Node and Redundant Computation Elimination

The last piece of graph-level cleanup is the most conceptually simple, but it's what makes every optimization above actually pay off in practice rather than being undermined by clutter left over from model export.

6.1 Genuinely unreachable subgraphs

Model export pipelines (from PyTorch, TensorFlow, or hand-authored ONNX) routinely leave behind computation that contributes to nothing the graph actually outputs — auxiliary training-only heads, debug taps, or branches left connected by an exporter that didn't prune them:

A graph optimizer performs a reachability analysis backward from the declared graph outputs: mark every node an output transitively depends on, and anything left unmarked is provably dead — it can be deleted with zero effect on any output, because by construction nothing downstream ever reads it. This is the same class of analysis a compiler performs for dead-code elimination in ordinary programs, applied to a dataflow graph instead of a control-flow one.

6.2 Redundant identity-like nodes

A second, narrower case: nodes that are logically necessary in some representation of the graph (often inserted by an exporter to make shapes or types explicit) but are semantically no-ops once the surrounding context is known:

X → Identity → Y      (Identity contributes nothing; replace with X → Y directly)
X → Reshape(same shape) → Y   (Reshape to the tensor's existing shape is a no-op)

ONNX Runtime's basic optimization level explicitly includes elimination passes for exactly this category — Identity, redundant Slice, Unsqueeze, and Dropout nodes (Dropout is a training-only regularization operator that becomes a pure passthrough at inference time, since there's no "drop" without gradient-based training happening).

6.3 Why order matters: elimination interacts with folding and fusion

Dead-node elimination is usually run both before and after other passes, because each optimization can create new dead code for the next pass to clean up. Folding a Conv-BatchNorm pair (Section 5) leaves the original, now-unused BatchNorm parameter tensors as orphaned constants with no consumer — dead-node elimination is what actually removes them from the compiled artifact rather than leaving them to bloat the model file and, on some backends, still occupy a memory allocation slot. This is the standard reason production graph-optimization pipelines (ONNX Runtime's included) run their passes to a fixed point — repeating basic optimizations until a pass produces no further change — rather than as a single linear sweep: fusion enables folding opportunities, folding creates dead nodes, elimination removes them, and the cleanup can occasionally expose yet another fusion or folding opportunity that was previously obscured.

7. Putting the Whole Pipeline Together

None of these four transformations live in isolation inside a real compiler. A representative ordering, close to what ONNX Runtime's tiered optimization levels and TVM's Relay pass pipeline both approximate:

1. Dead-node / redundant-node elimination   (clean up exporter artifacts first)
2. Constant folding                          (collapse constant-only subgraphs, e.g. Conv+BN)
3. Operator fusion                           (weld remaining runtime-dependent chains together)
4. Layout selection + transform insertion    (choose NCHW/NHWC per backend, minimize transform count)
5. Tiling / schedule generation              (size the loop nest to the target's SRAM/cache budget)
6. Dead-node elimination again                (sweep up anything the above passes orphaned)

Every one of these steps is, in the end, in service of the same single number Lesson 1 introduced: arithmetic intensity. Layout selection and fusion reduce bytes moved for a fixed amount of useful compute. Tiling maximizes reuse of whatever bytes do get moved. Constant folding and dead-node elimination remove computation — and its associated memory traffic — that never needed to exist at runtime at all. A graph that has been through all six steps isn't doing different math than the naive, unoptimized version exported straight from a training framework — it is doing the same math, arranged so that the hardware's memory system stops being the bottleneck.