The Runtime: Operator Placement, Scheduling, and DMA Overlap

Part 1 of this lesson worked out exactly where every tensor lives — a static plan, computed once, that reduces allocation to a lookup table of fixed offsets. That plan answers where. It says nothing about when, or on which device. Turning a planned graph plus a memory plan into actual instructions dispatched to actual silicon is the runtime's job, and it is a surprisingly deep one: the runtime has to decide which processor runs each operator, keep a plugin architecture around that decision so the same graph can target wildly different hardware, order the operators so nothing runs before its inputs exist, and — on any system with more than one piece of silicon — figure out how to keep all of that silicon busy at once instead of taking turns. This part picks up exactly where Part 1 left off and follows the plan all the way to dispatch.

1. What the Runtime Actually Does

It helps to be precise about the boundary between the two halves of the system, because it's easy to conflate "the compiler decided X" with "the runtime does X" when in practice they are different programs running at different times.

Everything in Part 1 — liveness analysis, interference graphs, buffer coloring, in-place operator detection — happens once, ahead of the first inference, either at model-export time on a developer's workstation or at model-load time on the target device. The output of all of that work is inert data: a graph (the sequence of operators and their dependencies) and a memory plan (a table mapping every tensor to a fixed buffer offset). Neither of those artifacts does anything by itself.

The runtime is the piece that turns inert data into motion. On every single inference call, it has to:

model loading         — deserialize the graph + memory plan into an in-memory session
memory                — reserve the arena(s) the plan describes, once, at session start
execution providers   — route each operator to the hardware backend that will run it
kernel dispatch        — call the actual compiled kernel for each operator, in order
synchronization        — make sure an operator's inputs are actually ready before it runs
threading              — decide how many CPU threads run concurrently, and on what
device transfers       — move data across memory-domain boundaries when placement changes
profiling               — record timing so the next placement/scheduling decision can improve

Every item on that list is a decision, not a formality — and the four decisions that matter most for making an already-planned graph actually run fast on real, heterogeneous hardware are the four this part is organized around: placement (which device runs which operator), execution providers (the abstraction that makes placement pluggable across hardware vendors), scheduling (the order operators actually get dispatched in, including which ones can run concurrently), and overlap (keeping compute and data movement happening at the same time instead of one waiting on the other).

Notice that the runtime sits between a static artifact (the plan) and dynamic hardware (whatever chips are actually in the device this build happens to run on). That's precisely why placement, EPs, and scheduling all live in the runtime rather than being baked into the plan at compile time on many systems: the same exported model needs to run correctly whether it lands on a phone with an NPU, a server with a GPU, or a microcontroller with neither. Some compilers (TVM's AOT flow, for instance) push placement earlier, into the compiler, precisely to avoid paying this decision cost at every load — but the decision itself — which device gets which operator — is a runtime-shaped problem no matter which stage in the pipeline actually resolves it.

2. Operator Placement Across Heterogeneous Hardware

Operator placement is the question of which physical device executes each node in the graph, and it only becomes interesting the moment a graph spans more than one kind of hardware. On a CPU-only deployment there's nothing to decide — every operator runs on the one processor available. The instant an NPU, GPU, or DSP enters the picture, placement becomes a real optimization problem, because different devices support different operator sets, and the boundary between "supported here" and "not supported here" rarely lines up cleanly with the natural boundaries of the model.

2.1 A Worked Example: The Conv → Unsupported-Op → Conv Sandwich

Take a small, entirely realistic three-operator slice of a larger model:

        Input (int8, NPU-native layout)
          │
          ▼
      [ Conv1 ]   — quantized int8 convolution, NPU-supported
          │
          ▼
     [ Data-dependent Gather/NMS-style op ]  — control flow driven by
          │                                     runtime tensor VALUES,
          │                                     not supported on the NPU
          ▼
      [ Conv2 ]   — quantized int8 convolution, NPU-supported
          │
          ▼
        Output

Conv1 and Conv2 are both good NPU citizens: fixed shapes, int8 weights and activations, no data-dependent branching — exactly the kind of operator a quantized accelerator's fixed-function convolution engine was built for (see Lesson 4's treatment of quantization for why int8 conv maps so cleanly to accelerator silicon). The middle operator is not. Suppose it's something like a non-maximum-suppression step or a data-dependent gather, where the number of elements or the indices being read depend on the values computed at runtime rather than being knowable from shape alone. Fixed-function NPU pipelines are built around static, shape-known execution — data-dependent branching is exactly the kind of thing they don't support, so this operator has to fall back to the CPU, which can execute arbitrary control flow.

That single unsupported operator forces the runtime into a placement decision that looks obviously correct in isolation and is expensive in aggregate:

   NPU: Conv1
    │
    ▼  device transition — cross out of NPU memory domain
   CPU: Gather/NMS-style op
    │
    ▼  device transition — cross back into NPU memory domain
   NPU: Conv2

Each arrow labeled "device transition" is not free, and it is not just a copy. A production NPU typically executes out of its own tightly-coupled local memory (an SRAM scratchpad the accelerator can address directly), not the same address space the CPU sees — that's precisely the DRAM/SRAM split this post returns to in Section 5. Crossing that boundary in either direction usually bundles together several distinct costs:

• A memory-domain copy. The tensor produced by Conv1 lives in NPU-local memory; the CPU op needs it somewhere the CPU can address. Some SoCs support direct load/store across the boundary; many require an explicit DMA or memcpy.
• A layout conversion. NPUs frequently want activations in a blocked or tiled layout optimized for their internal datapath; CPU kernels usually assume plain row-major. A layout mismatch across the boundary means the copy isn't a straight memcpy — it's a repack.
• A dtype conversion. If the CPU operator's reference implementation only exists in floating point, the int8 tensor has to be dequantized on the way out and requantized on the way back in — two more passes over the data, each with its own scale/zero-point arithmetic.
• A synchronization handshake. This is the one that's easy to forget and often dominates. The CPU cannot start reading Conv1's output until the NPU has actually finished writing it and has signaled that fact — typically via an interrupt, semaphore, or memory-mapped completion flag. That handshake carries a largely fixed latency that doesn't shrink just because the tensor being handed over is small.

This is exactly the mechanism Apache TVM's device-placement runtime formalizes directly: when a compiled graph is annotated so that different nodes target different devices, TVM's graph runtime automatically inserts a special __copy operator at every boundary where the device changes between a producer and its consumer, and that inserted copy is a real, scheduled operator with real cost — not a free relabeling. A device-placement decision that looks like "just run this one op on CPU" silently expands, mechanically, into "insert a copy operator into the graph, twice."

2.2 Quantifying the Boundary Cost

Put rough, representative numbers on the three-operator sandwich to see how badly a single unsupported operator can distort the picture, even when every individual kernel is fast:

Total wall-clock time: 435 microseconds. Time actually spent computing something the model needs (the two convs plus the gather): 90 microseconds — about 21% of the total. The remaining 79% is the fixed cost of crossing a device boundary twice, and the handshake latency alone (300 of the 345 transition microseconds) dwarfs the actual byte-copying cost. This is the concrete, arithmetic version of the claim ONNX Runtime's own documentation makes in prose: incomplete accelerator operator coverage, and the CPU fallback it forces, can produce substantial performance losses even when every kernel involved is individually well-optimized. Two fast convolutions and one fast CPU op combine, through nothing but the cost of switching devices twice, into a model an order of magnitude slower than its compute would suggest. That's the load-bearing inequality this whole section rests on:

end-to-end latency  >  sum of individual kernel latencies
 
               (the gap is exactly the device-transition cost,
                and it can dominate the total)

The practical response a placement pass takes to this is straightforward once the cost is visible: don't just ask "which device runs fastest for this operator in isolation," ask "which placement minimizes the number of device transitions along the graph's critical path." Sometimes that means deliberately running an operator on a slower device — including, in cases like small dequantize-copy-quantize sandwiches, choosing to keep an otherwise-CPU-preferred operator on the accelerator via a slower fallback kernel — purely to avoid paying two transition costs to save on one kernel's compute time. Placement is a graph-level optimization, not an operator-level one, which is exactly why it has to live in the runtime, with visibility into the whole graph, rather than being decided kernel-by-kernel.

3. Execution Providers: The Plugin Architecture That Makes Placement Possible

Section 2 described placement as a decision the runtime makes. It's worth being equally precise about how a runtime can even ask the question "which devices are available, and what can each of them run?" without hard-coding a different code path for every vendor's accelerator. The answer, in every major production runtime, is a plugin abstraction — ONNX Runtime calls it an execution provider (EP), TVM calls the equivalent mechanism a codegen target or BYOC backend, but the shape of the idea is the same everywhere.

Each execution provider is, functionally, a self-contained answer to two questions: which nodes in this graph can I run? and given that I claimed them, how do I run them? ONNX Runtime's session initialization calls a GetCapability() method on every registered EP, in priority order; each EP inspects the graph and reports back the specific nodes (or fused subgraphs) it's able to execute, and the runtime assigns those nodes to that EP, removes them from further consideration, and repeats the process with whatever EPs remain in the priority list — usually ending with the CPU EP, which supports the full ONNX operator set and therefore acts as the universal fallback that guarantees the graph as a whole is always executable, even when every hardware accelerator on the device declines every operator. This is the exact machinery behind Section 2's NPU → CPU → NPU sandwich: the NPU-flavored EP's GetCapability() call claims Conv1 and Conv2, declines the data-dependent op, and the CPU EP's GetCapability() call picks up what was declined. Partitioning the graph this way — instead of trying to compile one monolithic backend that understands every operator on every device — is precisely what lets the same ONNX file run unmodified on a laptop CPU, a datacenter GPU, and a phone's NPU: the graph representation never changes, only which EPs are registered and what they each claim.

3.1 A Comparison of Real Execution Providers

ONNX Runtime ships (or supports third-party) execution providers spanning essentially every category of inference hardware in production today. A few, chosen to span the range from general-purpose fallback to narrow, highly specialized accelerator:

The pattern across every row is identical to what Section 2 already worked out by hand: each EP claims exactly what it can execute well, declines the rest, and a lower-priority EP — ultimately the CPU EP — picks up the remainder. The EP abstraction doesn't eliminate device-transition cost; it's precisely what makes device-transition cost a visible, first-class scheduling problem instead of an implementation detail buried inside one monolithic backend. Once placement is expressed as "partition the graph among registered EPs," the runtime has exactly the information Section 2's cost table needs: which contiguous stretches of the graph share a device (no transition needed) and which node-to-node edges cross an EP boundary (a transition, with a real cost, has to happen).

4. Scheduling: Topological Order, Not Just a List

Placement answers which device. Scheduling answers in what order, and the honest answer for any non-trivial graph is: there isn't a single correct order, there's a whole family of valid orders, and the runtime's job is to pick one that exploits whatever concurrency the graph's dependency structure actually permits.

The graph a runtime executes is a directed acyclic graph — every operator has zero or more inputs it depends on, and it cannot run until all of them are ready. Any ordering of operators that never places a node before its dependencies is a valid topological order, and in general a DAG admits many of them. The scheduling question that actually matters for performance is: among all valid topological orders, which one — or which concurrent execution, since a strict ordering discards information that a smart scheduler shouldn't throw away — finishes fastest given the hardware actually available?

4.1 A Worked DAG: Independent Branches vs. a Dependent Chain

Compare two small subgraphs that look superficially similar — both have three operators — but have opposite scheduling properties.

In Case 1, A and B have no edge between them in the dependency graph — nothing A writes is read by B, and vice versa — so a scheduler is free to run them concurrently on separate devices (or separate threads on the same device), and only C needs a synchronization point that waits for both to complete. In Case 2, every operator's only input is the previous operator's only output; the dependency chain has zero width at every step, so there is exactly one valid schedule — strictly sequential — regardless of how many CPUs, GPUs, or NPUs the system has sitting idle. This is the same fact Amdahl's Law states about parallel programs in general, applied at the level of a single inference graph: the parts of a computation with no independent branches to exploit set a hard floor on how much concurrency is available, no matter how much hardware you throw at the problem.

4.2 Execution Streams and Synchronization Points

Realistic runtimes express "run these concurrently" using execution streams — independent, ordered queues of work, each with its own device or hardware queue, where operators within one stream execute strictly in order but operators in different streams have no ordering constraint between them except at explicit synchronization points. Case 1's schedule, expressed on two streams (say, one for a CPU-favored branch and one for an NPU-favored branch):

Time →                    0    1    2    3    4    5    6
Stream 0 (CPU):  A        [████████████]
Stream 1 (NPU):  B        [██████████████████████]
                                                    │
                                          sync point: C waits for
                                          BOTH streams to reach here
                                                    ▼
Stream 0/1:      C                                 [██████]

A finishes at t=3; B, running concurrently on the other stream, finishes at t=5. C cannot start until the later of the two — t=5 — because it genuinely needs both inputs; the wall-clock cost of the branch is max(duration(A), duration(B)) rather than duration(A) + duration(B)), which is exactly the saving concurrent scheduling buys whenever the dependency graph has real width to exploit. This is also precisely the shape of the "CPU/NPU overlap" pattern introduced in Section 5 below — preprocessing running on a CPU stream while a previous batch's inference runs concurrently on an NPU stream is Case 1's pattern applied to a full pipeline instead of a three-node subgraph.

Case 2's chain has no equivalent diagram worth drawing — every operator occupies the same single stream, one after another, and the only thing a scheduler can meaningfully optimize is which device that one stream targets, not how many streams to use. Real schedulers exploit this distinction directly: ONNX Runtime's session options expose an explicit sequential-versus-parallel execution mode, and even in its default sequential mode it maintains separate intra-op and inter-op thread pools specifically so independent branches like Case 1's A/B pair can be dispatched to different threads without the runtime having to fall back to strict, single-threaded topological order.

5. CPU/NPU Overlap and DMA

Section 4's A/B branch pattern generalizes into something more valuable than a one-time speedup on a single graph: if a pipeline processes a stream of inputs — video frames, audio chunks, tiled activations too large to fit in local memory at once — the "independent branch" doesn't have to be two different operators in the same graph. It can be two different stages of the pipeline itself, running one input behind another. That's the pattern this section works out in full, with the arithmetic, because it's the single highest-leverage optimization available once a model has moved past "make one kernel fast" and into "keep the whole system busy."

5.1 DMA, in One Paragraph a Firmware Engineer Already Believes

Direct Memory Access hardware exists to solve exactly one problem: without it, moving a block of bytes from one memory to another requires the CPU to sit in a loop, issuing a load and a store for every single word, unable to do anything else until the last byte lands. A DMA engine is a small, dedicated piece of hardware that can perform that same block transfer autonomously — the CPU (or NPU, or DSP) programs it with a source address, a destination address, and a length, then goes back to doing something else entirely, and the DMA engine raises an interrupt or sets a completion flag once the transfer is done. The core doing compute and the engine doing the transfer are, from that point until completion, two genuinely independent pieces of hardware making progress at the same time. Anyone who has wired an ADC or an I2S peripheral to fill a buffer over DMA on a microcontroller has already internalized the payoff of this pattern without needing to think about neural networks at all: the transfer happens "for free," in the sense that it costs no CPU cycles, provided the compute core has something useful to do with the time the transfer takes.

That is the entire mechanism behind CPU/NPU overlap. An NPU very often can't operate directly on data sitting in main DRAM — it has its own small, fast local memory (an SRAM scratchpad, a tightly-coupled memory, whatever a given vendor calls it), and getting data into that local memory is itself a DMA transfer, separate from and concurrent with whatever compute the NPU's math units are doing on data that's already local. This is precisely how Arm's Ethos-U NPU line is documented to operate: a DMA engine built into the accelerator copies data from shared system memory into the NPU's local memory ahead of when it's actually needed, so that the compute pipeline is fed continuously instead of stalling on every tile boundary waiting for a transfer that hasn't been started yet.

5.2 The Naive Serial Pipeline

Take a small, concrete workload: a tensor too large to fit in the NPU's local memory at once, so it's processed in N = 6 tiles. Each tile requires a DMA transfer in (duration d) before the NPU can compute on it (duration c). The naive way to write this loop is exactly the way anyone writes a first-pass implementation:

for tile in 1..N:
    DMA-load tile into the (single) local buffer
    compute on that buffer
    (buffer is now free — loop back and overwrite it with the next tile)

With a single buffer, the DMA transfer for tile i+1 cannot start until compute on tile i has finished and released the buffer — the DMA engine and the compute core are never doing useful work at the same instant. Take d = c = 4 ms (deliberately equal, to isolate the best case for overlap first) and draw the timeline:

Naive serial pipeline — d = 4 ms, c = 4 ms, N = 6 tiles
 
Time (ms):  0    4    8    12   16   20   24   28   32   36   40   44   48
DMA:        [L1] .    [L2] .    [L3] .    [L4] .    [L5] .    [L6] .
Compute:    .    [C1] .    [C2] .    [C3] .    [C4] .    [C5] .    [C6]
 
Total time = N × (d + c) = 6 × (4 + 4) = 48 ms

Every millisecond of DMA time is a millisecond the compute core is provably idle, and every millisecond of compute time is a millisecond the DMA engine is provably idle. Across the whole run, exactly half the timeline is wasted on each side — the DMA engine's whole reason for existing, the ability to run independently of the compute core, is being thrown away by a loop structure that never gives it the chance.

5.3 Double Buffering: Overlapping DMA With Compute

The fix needs exactly one more buffer. With two local buffers (A and B, ping-ponging), the DMA engine can load tile i+1 into whichever buffer isn't currently being read by the compute core working on tile i:

for tile in 1..N:
    DMA-load tile into buffer[tile % 2]        # loads into the OTHER buffer
    (concurrently:) compute on buffer[(tile-1) % 2]   # reads the buffer
                                                       # already loaded

Simulated explicitly, step by step, with the same d = c = 4 ms and N = 6:

Double-buffered pipeline — d = 4 ms, c = 4 ms, N = 6 tiles
 
Time (ms):  0    4    8    12   16   20   24   28
DMA:        [L1][L2][L3][L4][L5][L6]
Compute:         [C1][C2][C3][C4][C5][C6]
 
  t=0-4:   DMA loads tile1 -> buffer A                (pipeline fill,
                                                         nothing to compute yet)
  t=4-8:   compute tile1 (reads A)  ||  DMA loads tile2 -> buffer B
  t=8-12:  compute tile2 (reads B)  ||  DMA loads tile3 -> buffer A
  t=12-16: compute tile3 (reads A)  ||  DMA loads tile4 -> buffer B
  t=16-20: compute tile4 (reads B)  ||  DMA loads tile5 -> buffer A
  t=20-24: compute tile5 (reads A)  ||  DMA loads tile6 -> buffer B
  t=24-28: compute tile6 (reads B)                     (pipeline drain,
                                                         nothing left to load)
 
Total time = 28 ms

Every DMA load from tile 2 onward happens entirely inside the shadow of the previous tile's compute — the only DMA transfer that isn't hidden is the very first one, which has nothing to overlap with because the pipeline hasn't started yet. That single unhidden transfer is the "pipeline fill" cost, and it's unavoidable: the very first tile has to be resident in local memory before compute can begin on anything.

5.4 The Arithmetic: When Overlap Helps a Lot, and When It Barely Helps

Generalize the two timelines above into formulas, then check both against the worked numbers.

Plugging in the equal case, d = c = 4 ms, N = 6:

T_naive  = 6 × (4 + 4) = 48 ms
T_double = 4 + 6 × max(4, 4) = 4 + 24 = 28 ms
 
reduction = (48 - 28) / 48 = 20 / 48 ≈ 41.7%

That matches the hand-simulated Gantt diagrams exactly — the formula isn't an approximation here, it's the same arithmetic written compactly. The natural next question is: does overlap always buy roughly 40-50%, or does that number depend on the relationship between c and d? Take a second case where compute dominates DMA — c = 6 ms, d = 2 ms (same N = 6, and note c + d = 8 ms either way, so the naive total is unchanged):

T_naive  = 6 × (6 + 2) = 48 ms
T_double = 2 + 6 × max(6, 2) = 2 + 36 = 38 ms
 
reduction = (48 - 38) / 48 = 10 / 48 ≈ 20.8%

Roughly half the benefit of the balanced case, for the same total naive runtime. The reason is visible directly in the simulation: whenever d is less than c, the DMA engine finishes loading the next tile well before compute on the current tile is done and then simply sits idle for the remainder of that slot — there's nothing left to hide, because the transfer was already fully hidden. Overlap can only reclaim time that was genuinely wasted in the naive version, and in the naive version the wasted time on each side is exactly the smaller of c and d per tile. That gives a clean closed form for the best-case reduction as N grows large enough that the one-time pipeline-fill cost becomes negligible:

                    min(c, d)
reduction  →  ───────────────       as N → ∞
                    c + d

Check both cases against this limit: balanced case, min(4,4) / (4+4) = 4/8 = 50% — the finite-N result of 41.7% is approaching that limit from below, and the gap is entirely the one unhidden pipeline-fill transfer, which matters proportionally less as N grows. Compute-dominated case, min(6,2) / (6+2) = 2/8 = 25% — again, the finite-N result of 20.8% sits just under the limit for the same reason. The formula also explains the two extremes worth naming explicitly: when c and d are perfectly balanced, the reduction limit hits its maximum possible value of 50% — overlap can, at best, cut a balanced pipeline's time almost in half, never more, because the faster stage's time was never wasted to begin with, only the slower stage's redundant waiting was. When either stage dominates the other overwhelmingly (d much less than c, or c much less than d), the reduction limit collapses toward zero — there's a smaller and smaller amount of genuinely wasted time left for double buffering to reclaim, and the pipeline's total time becomes governed almost entirely by whichever single resource is the bottleneck, which is a problem double buffering was never going to solve in the first place (that's a workload-balancing problem, not a scheduling problem).

The practical takeaway for a runtime's scheduler: double buffering (and its natural extension, deeper pipelining with three or more buffers, sometimes called triple buffering, for systems where DMA latency is more variable) is worth the extra buffer memory almost unconditionally, but the payoff to expect from it should be estimated from the actual ratio of transfer time to compute time on the target hardware, not assumed to be "roughly half" by default. A workload where DMA is a rounding error next to compute time gains little from overlap and more from making the compute itself faster; a workload where the two are closely matched — which, not coincidentally, is exactly the regime a well-tuned tiling scheme from Lesson 4 tends to land in, since a tile size is very often chosen precisely to balance compute time against the time needed to keep it fed — is exactly where overlap earns its keep.

6. How Real Runtimes Do This

6.1 ONNX Runtime: Execution Providers, Partitioning, and Sequential vs. Parallel Execution

Sections 2 and 3 already described ONNX Runtime's EP model in the abstract: GetCapability() calls in priority order, graph partitioning, CPU EP as universal fallback. On the scheduling side, ONNX Runtime additionally exposes an explicit execution-mode setting on its session options: sequential execution, the default, still uses separate intra-op and inter-op thread pools so independent nodes can be dispatched concurrently without the runtime abandoning a single coherent schedule, while parallel execution mode relaxes ordering further for graphs with substantial independent width. Both modes are the productionized version of Section 4's A/B-branch pattern — the runtime is looking at the same dependency graph this post drew by hand and deciding, node by node, whether a synchronization point is actually required before the next op can be dispatched.

6.2 Arm Ethos-U: A DMA Engine Built Into the NPU

The Ethos-U line of NPUs, designed to pair with Arm Cortex-M microcontrollers, builds the DMA engine described in Section 5.1 directly into the accelerator rather than treating it as a separate system peripheral: the NPU's own DMA hardware moves data from shared system memory into its local SRAM ahead of when the compute pipeline needs it, and Arm's own architecture documentation describes inference workloads on the device as being broken into smaller jobs specifically so this data movement can be scheduled and overlapped with compute rather than executed as one blocking transfer per inference. That's Section 5.3's double-buffered ping-pong pattern, implemented in the accelerator's own hardware scheduler instead of software — the same idea, one level closer to the silicon.

6.3 TVM: device_copy and the Bring-Your-Own-Codegen Boundary

TVM's heterogeneous execution support formalizes Section 2's device-transition cost as an explicit, first-class graph operation rather than an implicit side effect. When a Relay graph is annotated so that different subgraphs target different devices — through its device-annotation API or through its Bring-Your-Own-Codegen (BYOC) framework, which lets a hardware vendor register a custom backend code generator for whatever subgraph pattern their accelerator can handle — TVM's compilation pass inserts an explicit device_copy (surfaced in the compiled graph runtime as a special __copy operator) at every edge where the producer and consumer are scheduled to different devices. That inserted node is scheduled and costed like any other operator in the graph, which is exactly the discipline Section 2.2's cost table argued for by hand: a device boundary shouldn't be a free relabeling hidden inside the runtime, it should be a visible, accounted-for line item the scheduler can reason about and, where possible, minimize.

Further Reading

• ONNX Runtime, "Execution Providers" — the authoritative overview of the EP abstraction, GetCapability()-based graph partitioning, and provider priority/fallback ordering.
• ONNX Runtime, "NNAPI Execution Provider" — a concrete example of an EP built as a unified abstraction over heterogeneous mobile hardware (CPU/GPU/NPU on Android).
• Arm, "ML Developers Guide for Cortex-M Processors and Ethos-U NPU" — Ethos-U's hardware architecture, including its built-in DMA engine and its role in overlapping data movement with compute.
• Apache TVM, "External Library Dispatch (BYOC)" — the Bring-Your-Own-Codegen framework, device annotation, and the device_copy/__copy mechanism inserted at heterogeneous-execution boundaries.
• Embedded.com, "Using Direct Memory Access Effectively in Media-Based Embedded Applications, Part 3" — DMA fundamentals and double-buffered/circular transfer patterns from an embedded-systems perspective.
• Wadix Technologies, "Producer-Consumer Made Simple: Double Buffering Explained" — an accessible walkthrough of the producer/consumer double-buffering pattern this post's pipeline-timing arithmetic is built on.

That closes Lesson 5. Between memory planning and the runtime, the graph now has a fixed place for every tensor to live and a working strategy for getting every operator dispatched, placed, and overlapped as efficiently as the hardware allows — the compile-time and dispatch-time halves of "make one inference fast." Lesson 6 turns to a different axis entirely: what happens when the question isn't "how fast is one inference" but "how many inferences per second," and why the batch size that minimizes latency is very often not the batch size that maximizes throughput.