Latency vs Throughput and the Batch Size Tradeoff

Lesson 5 established that a compiled, scheduled runtime can execute a single graph efficiently on a single input. Lesson 6 asks the question that turns a runtime into a server: how many requests should it execute at once, and who has to wait while it decides?

1. Latency and Throughput Are Not the Same Number

1.1 Two definitions that get conflated constantly

Latency is the time from when one request arrives to when its result is available. For a single inference call, that's the wall-clock span from "input tensor handed to the runtime" to "output tensor returned" — everything Lessons 1 through 5 spent time optimizing lives inside this number. It is a property of one request.

Throughput is the rate at which a system completes requests, measured in requests per second (or, in some contexts, tokens per second, images per second, samples per second). It is a property of the system over an interval, not of any single request.

The conflation happens because in the simplest possible server — one that accepts a request, runs it to completion, and only then accepts the next one — the two are reciprocals by construction: if every request takes L seconds and the server never does two things at once, it completes 1/L requests per second, full stop. That identity is where the intuition "just invert latency to get throughput" comes from, and it is also exactly the assumption that batching breaks.

1.2 A worked example where they diverge

Consider a small image classification model running on a GPU-backed server. Measured in isolation, a single forward pass — one image in, one prediction out, nothing else happening on the device — takes 10 milliseconds. Naively inverting that number suggests the server can handle 100 requests per second (1 / 0.010).

Now suppose the server groups incoming requests into batches of 20 before running them, and a batch of 20 images takes 25 milliseconds to execute — far less than 20 × 10ms = 200ms it would take to run them one at a time, because the GPU's compute units were sitting mostly idle during each solo 10ms pass (Section 2 derives exactly why). If the server can keep a steady stream of batches moving through the device, its throughput is:

Throughput = batch_size / batch_execution_time
           = 20 / 0.025 s
           = 800 requests/second

That's 8x the naive 1/L estimate of 100 req/s. But look at what happened to any individual request's latency. A request that arrives just after a batch has started filling has to wait for the rest of the batch to accumulate, then wait the full 25ms of batch execution, before its result comes back. Depending on how long the queueing wait was (Section 3 derives this precisely), that request's true end-to-end latency could easily be 40, 80, or 150 milliseconds — four to fifteen times worse than the solo-request 10ms figure, even though the system's throughput went up by 8x in the same change.

This is the central fact this chapter builds on: once batching is in play, throughput and latency are two separate curves as a function of batch size, and they move in opposite directions. Throughput improves with batch size (Section 2 derives why, precisely). Latency degrades with batch size (Section 3 derives why, precisely). Neither is the reciprocal of the other, and a server's job is to pick a point on that tradeoff, not to optimize either number in isolation.

1.3 Batching is not the only lever — a note on pipelining

The 800 req/s figure in Section 1.2 folded together two distinct mechanisms that are worth separating, since production systems use both and it's easy to credit the wrong one. Batching, the subject of this chapter, groups multiple requests into a single kernel invocation so the hardware processes them together, exactly as derived in Section 2. Pipelining is a different mechanism: overlapping the stages of different requests in time, so that while one request's batch is executing on the compute units, the next batch is already being copied onto the device, and the batch after that is still being assembled in the queue. Neither stage waits idle for the others to fully finish before starting its own next unit of work.

Without pipelining (stages run strictly in sequence):
  [ fetch batch 1 ][ run batch 1 ][ return batch 1 ][ fetch batch 2 ][ run batch 2 ] ...
 
With pipelining (stages overlap across batches):
  [ fetch batch 1 ][ run batch 1 ][ return batch 1 ]
                   [ fetch batch 2 ][ run batch 2 ][ return batch 2 ]
                                    [ fetch batch 3 ][ run batch 3 ] ...

Pipelining raises throughput the same way instruction-level pipelining does in a CPU: it doesn't make any single batch execute faster, it hides one batch's non-compute overhead behind another batch's compute. It compounds with batching rather than substituting for it — a server pipelining its data transfers while also batching requests within each transfer gets both effects at once, which is part of why real serving throughput numbers can look larger than the batching arithmetic in Section 2 alone would predict. This chapter's derivations isolate the batching effect specifically, since that's the lever with the direct connection back to arithmetic intensity and the Roofline model; pipelining is a real and complementary optimization, but a mechanically different one, and mixing the two into a single unexplained throughput multiplier is a common source of confusion when reading serving-system benchmarks.

1.4 Why this isn't just a serving detail

It's worth being explicit about why this matters enough to be its own lesson rather than a footnote to Lesson 5. Every optimization in Lessons 1 through 5 — arithmetic intensity, operator fusion, kernel scheduling — was about making one execution of the graph as fast as possible. That work is a necessary input to both latency and throughput, but it doesn't decide the tradeoff between them. A perfectly optimized single-request kernel still faces the batching decision this chapter covers, and a poorly optimized one will just shift both curves worse without changing their shape. The batch-size lever sits on top of everything the earlier lessons built, and it is the first decision in this series that is fundamentally about the workload (how many requests, how they arrive, what the product needs) rather than about the kernel.

2. Why Batching Increases Throughput: From GEMV Back to GEMM

2.1 Recap: batch-1 decode is memory-bound, and it isn't a tuning problem

Lesson 1, Section 7 derived this precisely, so it's worth restating the conclusion rather than the derivation: for a linear layer running at batch size 1, the operation is a matrix-vector multiply (GEMV) — one activation vector against a weight matrix — and its arithmetic intensity works out to

AI(batch=1) = 2 / S

where S is the byte size of one element (4 for fp32, 2 for fp16/bf16, 1 for int8). Every dimension-dependent term cancels out of that ratio. It does not matter how wide or deep the layer is — batch-1 arithmetic intensity is stuck at a small constant, typically 1–4 FLOP/byte, which sits roughly two orders of magnitude below the ridge point of essentially any modern accelerator (Lesson 1 computed ridge points around 150–160 FLOP/byte for both an edge NPU and a datacenter GPU). The weight matrix has to be streamed from memory once per single-vector multiply, in full, because there is only one activation vector to reuse each loaded weight against. No amount of clever kernel scheduling changes that; it's a structural property of the operation, not an implementation defect.

2.2 Batching turns the GEMV back into a GEMM

The fix is exactly the one Lesson 1 previewed and this chapter now works through in full: stack multiple requests' activation vectors into a matrix before running the layer. A [1, d_in] vector against a [d_in, d_out] weight becomes a [B, d_in] matrix against the same [d_in, d_out] weight — the operation is now a genuine matrix-matrix multiply (GEMM), and the weight matrix, still loaded from memory exactly once, gets reused B times instead of once. Re-deriving arithmetic intensity with the batch dimension in:

FLOPs(B) = 2 × B × d_out × d_in
 
Bytes(B) ≈ weight bytes (dominant, loaded once)
         +  B × d_in × S    (input activations, B vectors)
         +  B × d_out × S   (output activations, B vectors)
 
Bytes(B) = d_out × d_in × S  +  B × (d_in + d_out) × S

For any batch size small enough that the activation term stays negligible next to the weight term — true for most practical batch sizes, since weight matrices are typically thousands of times larger than a single activation vector — this simplifies to the clean result from Lesson 1:

AI(B) ≈ 2 × B × d_out × d_in / (d_out × d_in × S)
       = 2B / S

Arithmetic intensity scales linearly with batch size. This is not a heuristic or an empirical observation about serving systems — it's the same mechanism as the tiled-matmul result from Lesson 1, Section 2.2 (AI_ideal(N) = N/6, growing linearly with matrix dimension): batching literally makes the matrix bigger along the reuse-enabling dimension, and arithmetic intensity is, at its core, a measure of how many times each loaded byte gets reused before being discarded. A bigger batch means more reuse per byte, full stop.

2.3 Worked numeric example: a real feed-forward layer at batch 1, 8, 32, 128, 256

Take a concrete, representative layer — a feed-forward down-projection sized like those in a modern ~8B-parameter transformer: d_ff = 14336, d_model = 4096, fp16 weights (S = 2 bytes). This is exactly the operator shape underlying every decode step in autoregressive generation.

Weight matrix. W has d_model × d_ff = 4096 × 14336 = 58,720,256 elements, so at 2 bytes each:

Weight_bytes = 58,720,256 × 2 = 117,440,512 bytes ≈ 117.44 MB

FLOPs at batch B. Each output element is an inner product of length d_ff, giving 2 × d_ff FLOPs, times d_model output elements, times B requests:

FLOPs(B) = 2 × B × d_model × d_ff = B × 117,440,512

(The batch-1 FLOP count and the weight byte count come out numerically identical here — 117,440,512 either way — which is exactly the 2/S = 2/2 = 1 result from Section 2.1 showing up as a coincidence of fp16's element size, not a general law.)

Total bytes at batch B, including the small but non-zero activation traffic:

Bytes(B) = 117,440,512  +  B × (4096 + 14336) × 2
         = 117,440,512  +  B × 36,864

Plugging in batch sizes:

Batch `B`	FLOPs	Bytes	Arithmetic intensity (FLOP/byte)	Regime (edge NPU, ridge ≈ 156.25)
1	`117.44 MFLOP`	`117.48 MB`	`≈1.00`	deeply memory-bound
8	`939.52 MFLOP`	`117.74 MB`	`≈7.98`	memory-bound
32	`3.76 GFLOP`	`118.62 MB`	`≈31.68`	memory-bound
128	`15.03 GFLOP`	`122.16 MB`	`≈123.06`	memory-bound, close
256	`30.06 GFLOP`	`126.88 MB`	`≈236.96`	compute-bound

Solving AI(B) = AI* for the edge NPU's ridge point from Lesson 1 (AI* = 156.25 FLOP/byte) gives the exact crossing point:

117,440,512 × B = 156.25 × (117,440,512 + 36,864 × B)
B ≈ 164

Somewhere between batch 128 and batch 256, this specific layer crosses from memory-bound to compute-bound on this specific device — at roughly batch 164. Because the datacenter GPU's ridge point from Lesson 1 (≈153 FLOP/byte) is nearly identical, the crossing batch size is almost the same there too (≈161), which is exactly the point Lesson 1 made about ridge points being a ratio, not an absolute spec-sheet number: two wildly different pieces of silicon can demand a very similar batch size before a given kernel stops being bandwidth-starved.

This number — a batch in the low hundreds before this layer becomes compute-bound — is not a coincidence of the example; it's in the same range production LLM-serving systems actually target for decode batching, which is a useful sanity check that the derivation, not just the intuition, is doing real work here.

2.4 What "improved hardware utilization" concretely means

It's worth being precise about the mechanism, because "batching improves utilization" is often stated without the arithmetic-intensity chain underneath it. Two distinct effects are bundled into that phrase, and it's useful to separate them:

Amortized fixed overhead. Every kernel launch, every RPC round-trip into the serving framework, every host-to-device copy has a roughly constant cost regardless of how much data rides along with it. Spreading that fixed cost over 32 requests instead of 1 divides its per-request contribution by 32.
Higher arithmetic intensity, derived above. This is the deeper effect, and the one that actually moves a kernel's achieved FLOP/s on the Roofline chart from Lesson 1 — not just avoiding overhead, but genuinely reusing each loaded byte more times before it's discarded.

Both effects are real and both point the same direction, but the second one is why batching keeps paying off well past the point where launch overhead has become negligible: it is fundamentally changing which side of the Roofline bound the kernel sits on, exactly as Section 2.3 measured.

3. The Latency Cost of Batching: Queueing Delay

3.1 Setting up the model

Section 2 showed batching is, in principle, free extra throughput — same silicon, more reuse. It isn't free in wall-clock terms, because a batch has to fill before it can be dispatched, and filling takes time that depends on how fast requests arrive.

Model requests arriving as a stream with inter-arrival times that are independent and identically distributed with mean 1/λ (λ is the average arrival rate in requests/second — this covers both a steady deterministic stream and the more realistic Poisson-arrival case, since the derivation below only uses linearity of expectation over the inter-arrival times, not their exact distribution). A static batcher waits until exactly B requests have accumulated, then dispatches the batch for execution and starts accumulating the next one.

Label the requests in one forming batch 0, 1, …, B-1 in arrival order. Request 0 opens the batch; request B-1 is the one whose arrival completes it and triggers dispatch. Request i has to wait for B - 1 - i more arrivals after its own before the batch closes, and since each inter-arrival gap has expectation 1/λ:

E[wait_i] = (B - 1 - i) / λ

3.2 Deriving average and first-request wait

Average wait across the batch — averaging E[wait_i] over i = 0 … B-1:

E[wait_avg] = (1/B) × Σ_{i=0}^{B-1} (B - 1 - i) / λ
            = (1/B) × [(B-1) + (B-2) + … + 0] / λ
            = (1/B) × [B(B-1)/2] / λ
            = (B - 1) / (2λ)

Wait for the unluckiest request — the one that opens the batch (i = 0) has the longest expected wait of anyone in the batch:

E[wait_first] = (B - 1) / λ

This is an expectation, not a hard ceiling — with Poisson or otherwise bursty arrivals, any individual batch's opening request can wait considerably longer than (B-1)/λ if the stream happens to go quiet right after it arrives, since arrival processes have no upper bound on how long a gap can run. Treat (B-1)/λ as the steady-state average for that batch position, and Section 4's timeout mechanism as the actual fix for the unbounded tail.

3.3 Worked examples

Busy server: λ = 100 requests/second (mean inter-arrival 10ms), static batch size B = 32.

E[wait_first] = 31 / 100  = 0.31 s  = 310 ms
E[wait_avg]   = 31 / 200  = 0.155 s = 155 ms

Compare that to the layer's own execution time from Section 2.3 — a few milliseconds at most, even at batch 32. The queueing delay here is 30–100x larger than the compute it's protecting. A request landing in this server sees end-to-end latency dominated almost entirely by waiting for other people's requests to show up, not by any inference math.

Fast-arriving server: same batch size, λ = 10,000 requests/second (mean inter-arrival 0.1ms):

E[wait_first] = 31 / 10,000  = 0.0031 s = 3.1 ms
E[wait_avg]   = 31 / 20,000  = 0.00155 s = 1.55 ms

Same batch size, same batcher logic, a 100x difference in arrival rate — and the queueing delay drops by the same 100x factor, from a latency-dominating 155ms average down to a perfectly tolerable 1.55ms average, now comparable to or smaller than the batch's own compute time. Queueing delay is not a function of batch size alone; it's a function of the batch-size-to-arrival-rate ratio, B/λ. A batch size that's a reasonable throughput/latency tradeoff at one traffic level can be a badly wrong choice an order of magnitude later in the same day, as request volume rises and falls — a static, hand-picked batch size has no way to track that on its own.

3.4 The other cost: memory grows with batch size

Queueing delay is the latency-side cost of batching; there's also a resource-side cost that doesn't show up in either latency or throughput directly but bounds how far the batch-size lever can be pushed. Every tensor touched by the layer scales with B: activations, in the derivation above, contributed the B × (d_in + d_out) × S term to bytes moved, and that same memory has to be physically resident on the accelerator for the batch's duration — not just streamed past it. For a single dense layer this is a small fraction of the weight footprint, as Section 2.3's table showed. For a full transformer's autoregressive decode, the dominant batch-scaling memory cost is the per-sequence KV cache, which grows linearly in both batch size and sequence length simultaneously — large enough, in practice, to be the actual ceiling on how big a batch a given accelerator's memory can hold, often well before compute or bandwidth becomes the binding constraint. That mechanism is specific enough to deserve its own full derivation, which is exactly the subject of Lesson 6, Part 2.

4. The Production Answer: Adaptive Batching With a Timeout

4.1 Why a fixed batch size alone doesn't work

Section 3.3 already showed the problem: a static batcher that always waits for exactly B requests ties latency to whatever the arrival rate happens to be at that moment, with no upper bound when traffic is thin. If λ drops — a quiet period, an off-peak hour, a single low-traffic client — a batcher waiting for 32 requests to accumulate can stall indefinitely, or at least for an unacceptably long time, even though a batch of 3 or 4 would have been perfectly fine to dispatch immediately.

The fix used throughout real serving systems is dynamic (adaptive) batching: dispatch a batch either when it reaches its target size, or when a maximum wait time has elapsed since the batch started forming — whichever comes first — and send whatever has accumulated by then, even if it's smaller than the target.

Batch-forming timeline, target size = 8, timeout = 10 ms
 
t=0ms   request A arrives → batch opens, timeout clock starts
t=1ms   request B arrives   [A,B]
t=3ms   request C arrives   [A,B,C]
t=6ms   request D arrives   [A,B,C,D]
t=10ms  ── TIMEOUT reached, batch not full ──
        dispatch [A,B,C,D]  (size 4, not the target 8)
        new batch opens for the next arrival

Same target, same timeout, busier traffic
 
t=0ms   request A arrives → batch opens, timeout clock starts
t=1ms   B   t=2ms C   t=3ms D   t=4ms E
t=5ms   F   t=6ms G   t=7ms H → batch full (size 8) before timeout
        dispatch [A..H] immediately at t=7ms, timeout never fires

This single mechanism caps worst-case queueing delay at the timeout value, regardless of how slow arrivals get, while still capturing the full arithmetic-intensity win from Section 2 whenever traffic is heavy enough to fill the batch first.

4.2 Triton Inference Server's dynamic batcher

NVIDIA's Triton Inference Server implements exactly this pattern as its dynamic batcher, configured per-model in config.pbtxt. The two parameters that matter here are max_batch_size, the largest batch the batcher will assemble, and max_queue_delay_microseconds, the timeout: when a maximum- or preferred-size batch can't be formed from what has arrived, the batcher delays dispatch only as long as no request has been waiting longer than that configured value, after which it sends whatever it has, even below the target size. Triton's own documentation is explicit that dynamic batching is the single largest performance lever available for most models running on it, ahead of most kernel-level tuning — a direct real-world confirmation of the Section 2 mechanism.

4.3 TensorFlow Serving's batching parameters

TensorFlow Serving exposes the identical idea through a --enable_batching flag and a batching-parameters file setting max_batch_size, batch_timeout_micros, max_enqueued_batches, and num_batch_threads. The official guidance for batch_timeout_micros reflects Section 3's tradeoff directly: for latency-sensitive online serving it should typically be tuned to a few milliseconds — including zero, which works well for some workloads by dispatching as soon as anything is available — while bulk offline-style batch jobs, where nothing is waiting on the result in real time, can set it to a much larger value (seconds) purely to maximize batch fill and therefore throughput. Framed in this chapter's terms: TensorFlow Serving's own docs are recommending exactly the B/λ sensitivity from Section 3.3 be resolved in favor of latency for interactive traffic and in favor of arithmetic intensity for offline traffic.

4.4 Worked example: a mismatched timeout defeats itself

Return to the busy-server numbers from Section 3.3: λ = 100 requests/second, mean inter-arrival 10ms, target batch size 32. Set a timeout of 10ms.

In expectation, only λ × 0.010 = 1 request arrives within any 10ms window. The timeout will fire almost every time before the batch is anywhere close to full — dispatched batches will average roughly size 1–2, essentially back to unbatched execution. Per Section 2.3's table, arithmetic intensity stays pinned near AI ≈ 1, deep in the memory-bound region; almost none of the throughput benefit the batch size of 32 was chosen for is actually realized. The timeout, chosen without reference to the arrival rate, has silently overridden the batch-size decision.

Now raise the timeout to 100ms at the same arrival rate. Expected arrivals within the window become 100 × 0.1 = 10, giving batches that typically land around size 8–10 (occasionally larger, occasionally smaller, and capped at 32 if a burst fills it early) — arithmetic intensity around 8–10 per Section 2.3's table, a real if partial step toward the ridge point, at the cost of an average queueing wait now in the tens of milliseconds. Whether that's an acceptable trade depends entirely on the product's latency budget, which is exactly why this number is a tuning knob exposed to the operator rather than a constant baked into the runtime.

The general principle behind both worked numbers: the timeout should be set relative to the expected inter-arrival time 1/λ, not chosen as an arbitrary round number. A timeout much shorter than B / λ (the expected time to naturally fill the batch) makes the target batch size irrelevant — the timeout fires first almost every time. A timeout much longer than B/λ makes the timeout irrelevant — the batch fills before it matters, and all the timeout does is set an unused upper bound on tail latency during traffic dips.

4.5 Reading the knobs together

Put the two production systems' parameters side by side against the mechanisms derived in this chapter:

Parameter	Triton	TensorFlow Serving	What it controls (this chapter)
Target/preferred batch size	`preferred_batch_size`	`max_batch_size`	how far right on the AI axis a full batch pushes the kernel (Section 2)
Timeout	`max_queue_delay_microseconds`	`batch_timeout_micros`	the hard cap on `wait_first` from Section 3.2, overriding the unbounded queueing tail
Queue capacity	(implementation-specific)	`max_enqueued_batches`	backpressure once arrival rate exceeds the server's sustained processing rate

None of the three parameters can be chosen correctly in isolation. Batch size without a timeout reintroduces the unbounded-wait problem from Section 3.2. A timeout without regard to the arrival rate can silently cap the achievable batch size far below the target, as Section 4.4 showed numerically. This is, in miniature, the entire chapter: throughput (via arithmetic intensity) and latency (via queueing delay) are two ends of the same lever, and adaptive batching with a timeout is the mechanism production systems use to place that lever at a deliberately chosen point rather than wherever traffic happens to push it.

5. Edge vs Datacenter: Where Batch-1 Isn't a Choice

Everything in Sections 2 through 4 assumes there's a pool of concurrent requests to batch together in the first place. That assumption holds for a datacenter inference server fielding traffic from many independent clients, but it breaks down completely for a large class of edge and embedded systems, and it's worth being explicit about why, since it draws a sharp line through what the batch-size lever can and can't do.

Consider a wearable device streaming from a single onboard sensor:

There is exactly one stream of samples, produced sequentially by one piece of hardware, and exactly one consumer of each inference result — typically a real-time decision (trigger an alert, log an event, adjust a duty cycle) that has to happen before or shortly after the next sample arrives. There is no second, independent request arriving concurrently to batch this one with; λ in Section 3's terms is bounded by the sensor's own sampling rate, and waiting to accumulate a batch of 32 samples before running inference on any of them would mean holding up the very first sample's decision until 31 more had been collected — actively working against the application's latency requirement rather than trading against it. For this class of workload, batch size is fixed at 1 not as an unoptimized default but as a structural consequence of having a single, sequential input stream with a hard real-time consumer.

This is also why edge and datacenter inference optimization end up pulling on different levers, even when running comparable model architectures. A datacenter server has Sections 2 through 4 of this chapter available to it: it can spend queueing delay to buy arithmetic intensity, because it usually has many concurrent, latency-tolerant-enough requests to draw a batch from. An edge device generating one sample at a time from one sensor is stuck at the batch-1 GEMV arithmetic intensity derived in Section 2.1 — 2/S FLOP/byte, deep in the memory-bound region — with no batching escape hatch available at all. Its only remaining levers are the ones Lesson 1 spent its full length deriving: fusing operators to cut redundant DRAM round-trips, quantizing to shrink S and directly raise 2/S, and tiling to keep as much of the working set resident in on-chip SRAM as the model's structure allows. Batching is the throughput lever available when there's a queue to draw on; when there structurally isn't one, every remaining unit of arithmetic intensity has to come from the kernel itself.