Amdahl's Law, Latency Budgets, and Energy Efficiency

Lesson 9's profiling work told you exactly where a pipeline's milliseconds go, stage by stage, kernel by kernel.

This part asks what to actually do with that map.

Amdahl's Law turns a profile into a spending plan. Latency budgets turn a spending plan into a hard deadline. And on battery-powered hardware, milliseconds turn out to not even be the only currency that matters.

1. From Chips to Pipelines: Amdahl's Law for Inference

The companion Computer Architecture series already derived Amdahl's Law in full, in its original habitat: predicting the speedup from adding more cores to a CPU.

The derivation doesn't need repeating here.

What's worth restating is the shape of the result, because it's about to get reused in a context Gene Amdahl never had in mind.

Split any system's execution time into a part that responds to some optimization and a part that doesn't.

If a fraction p of total time is affected by a speedup of factor s, the whole system speeds up by:

Speedup = 1 / ((1 - p) + p/s)

The architecture series applied this to cores and vector lanes: p was "the parallelizable fraction of a program," s was "the number of processors."

Nothing in the math cares what p and s actually represent, though.

The law is a statement about any system built from stages, where you can only ever speed up the stages you touch, and the stages you don't touch keep running at their original speed regardless.

An inference pipeline is exactly that kind of system.

A typical embedded vision pipeline looks like this:

Each arrow is a stage boundary.

Each box has a measured latency, because Lesson 9 already taught you how to get one — instrument the pipeline, get a per-stage breakdown, and you have p for every stage in the system: its share of total wall-clock time.

Amdahl's Law then answers the question every optimization plan actually needs answered before a single line of kernel code changes: given a fixed amount of engineering time, which stage's p makes it worth optimizing at all?

This reframing matters because the intuitive instinct — "optimize whatever looks slow, or whatever's easiest to optimize" — is a category error.

A stage's absolute latency is not what determines the payoff of speeding it up.

Its share of the total is.

A stage can be conspicuously, embarrassingly slow in isolation and still be a bad place to spend a week, if it's a small enough slice of the whole pipeline.

2. Worked Example: Where Not to Spend Your Week

Take a concrete, representative embedded object-detection pipeline, quantized and running on an NPU, with a measured total latency of 40 ms per frame:

This is a realistic split for a conv-backbone detector.

The backbone dominates because it's where essentially all the FLOPs live — this is the same conclusion the Roofline model post would predict just from counting operations.

Postprocessing looks unreasonably expensive at 20% for something that's "just NMS," but NMS on unoptimized candidate boxes is notoriously branchy, memory-scattered, and a poor fit for the same SIMD/NPU hardware that makes the backbone fast.

That's exactly why it often survives as a slow, unglamorous CPU-side afterthought even in an otherwise well-optimized pipeline.

Now run three scenarios through Amdahl's Law.

Scenario A — spend a week making postprocessing 2× faster.

This is a believable outcome for a week of real effort: NMS is annoying to vectorize, box decode has awkward data dependencies, and getting a clean 2× typically means rewriting the candidate-filtering logic to avoid branch misprediction and cache-unfriendly access patterns.

Here, p = 0.20, s = 2:

Speedup = 1 / ((1 - 0.20) + 0.20/2) = 1 / (0.80 + 0.10) = 1 / 0.90 ≈ 1.111×

New total latency: 40 ms − 4 ms = 36 ms.

An 11% improvement, for a week of work on the hardest-to-vectorize stage in the pipeline.

Scenario B — spend a day getting the backbone 10% faster.

A 10% win on a quantized conv backbone is a modest, believable outcome for far less effort than Scenario A — better tiling, a fused activation, slightly improved cache reuse in the kernel.

p = 0.70, and "10% faster" means the stage now runs in 90% of its original time, i.e. s = 1/0.9 ≈ 1.111:

Speedup = 1 / ((1 - 0.70) + 0.70/1.111) = 1 / (0.30 + 0.630) = 1 / 0.930 ≈ 1.075×

New total latency: 40 ms − 2.8 ms = 37.2 ms.

A 7.5% improvement — slightly less than Scenario A's headline number, but for a fraction of the effort.

Already this should give pause: postprocessing "won" on paper, but it took a week to do it, and the backbone got most of the way there in a day.

Scenario C — spend that same week on the backbone instead, and get the same 2× you got on postprocessing.

This is the fair comparison: identical engineering budget, applied to the stage that actually dominates the profile.

p = 0.70, s = 2:

Speedup = 1 / ((1 - 0.70) + 0.70/2) = 1 / (0.30 + 0.35) = 1 / 0.65 ≈ 1.538×

New total latency: 40 ms − 14 ms = 26 ms.

A 54% improvement — roughly five times the total-latency gain of Scenario A, for the identical amount of engineering time.

There's a second, sharper way to see the same conclusion: compute each stage's ceiling.

The best-case speedup if that stage were made infinitely fast — s → ∞, so the term p/s vanishes entirely and the speedup collapses to 1/(1 - p).

Postprocessing's ceiling, at p = 0.20:

1 / (1 - 0.20) = 1.25×

Even deleting postprocessing outright — 0 ms, not just 2× faster — caps total speedup at 25%.

No amount of cleverness applied to that stage alone can ever beat that number, because the other 80% of the pipeline is still running at its original speed.

The backbone's ceiling, at p = 0.70:

1 / (1 - 0.70) = 3.33×

A perfect, zero-cost backbone would triple the pipeline's throughput.

Amdahl's Law, applied to a pipeline, is a statement about where the ceiling is before you've written a single optimization. A stage's fractional share of total time isn't just a hint about where to look — it's a hard mathematical bound on the best possible outcome of optimizing that stage in isolation, independent of how clever the optimization is. Profile first, because profiling is how you find p. Then let p decide where the week goes, not which stage looks most annoying, or most familiar, or most "obviously slow" in isolation.

This is precisely the connective tissue Lesson 9's profiling work was building toward.

A flame graph or a per-kernel trace answers "where does time go."

Amdahl's Law is the next sentence in that conversation — "given where time goes, what's the most it's worth fixing."

A profiler without this framing produces a list of slow things.

A profiler with this framing produces a ranked list of worthwhile things, which is a very different and much more useful artifact.

One caveat worth stating explicitly: this framing assumes stages are independent and additive, which is the common case for a straight-line inference pipeline but not universal.

If speeding up the backbone changes memory pressure enough to also speed up (or slow down) data transfer — say, by changing an intermediate tensor's layout or size — the stages are coupled, and the clean Amdahl accounting needs to be re-measured after the change, not just predicted from the old profile.

Treat the ceiling calculation as a planning tool for where to start, and re-profile after each real change to confirm the actual outcome — the same discipline Lesson 9 already established.

3. Latency Budgets: The Deadline Nobody Can Renegotiate

Amdahl's Law tells you how to spend a fixed amount of optimization effort.

It says nothing about whether the result is good enough.

That's a separate question, and it has a separate answer: the latency budget.

A latency budget is a hard external deadline the entire pipeline must fit inside, imposed by something outside the software — a human perception threshold, a physical sampling rate, a communication protocol's timing window.

It is not negotiable the way "make it 10% faster" is negotiable.

A voice assistant that responds in 250 ms instead of a 200 ms target isn't 25% less impressive — it's perceptibly broken.

Users experience response latency above roughly 200 ms as a noticeable lag rather than a snappy reply.

A camera pipeline that misses its frame deadline doesn't produce a slightly-late frame; depending on the buffering strategy, it drops a frame, or it backs up the whole downstream pipeline.

For an embedded sensor-fusion loop, the deadline comes from the sampling rate itself.

If the system samples at 100 Hz, one full cycle — read the sensor, process it, run inference, act on the result — must complete within 1/100 Hz = 10 ms, every single cycle, indefinitely, or the loop cannot keep pace with its own sensor and either falls behind or has to skip samples.

The discipline this creates is different from ordinary latency optimization.

Ordinary optimization asks "how can this be faster."

A latency budget asks "does every stage fit inside the deadline, together, at the same time, and what happens to the rest of the pipeline the moment one stage doesn't."

It converts a performance question into a scheduling and allocation problem.

You're not just chasing a number down — you're partitioning a fixed resource (time) across competing consumers (pipeline stages), the same way a memory planner partitions a fixed SRAM budget across competing tensors.

4. Worked Example: A 10 ms Sensor-Fusion Loop

Take a wearable running sensor fusion — combining accelerometer and gyroscope samples into an orientation or activity estimate — at 100 Hz.

The budget is 10 ms per cycle, and it has to be allocated across every stage the cycle actually does:

100 Hz sensor-fusion loop — hard budget = 10.0 ms / cycle
 
Sensor read (I2C burst, Fast Mode 400 kHz)     0.4 ms
Preprocessing (unit convert, LPF)              0.5 ms
Inference (on-device fusion/classifier)        6.0 ms
Actuation / output prep (haptic, BLE stage)    1.0 ms
Margin                                         2.1 ms
--------------------------------------------------------
Total                                         10.0 ms

Where does the 0.4 ms sensor-read figure come from?

It's worth deriving rather than asserting, because it's a good example of a budget line item that's easy to get wrong by an order of magnitude if you reason from vibes instead of the bus protocol.

I2C Fast Mode runs at 400 kHz, so one bit period is 1/400,000 Hz = 2.5 µs.

Each transferred byte on I2C costs 9 bit periods — 8 data bits plus one acknowledge bit — so roughly 9 × 2.5 µs ≈ 22.5 µs per byte, before counting start/stop condition and register-address overhead.

A typical 6-axis IMU burst read (3-axis accel + 3-axis gyro, 2 bytes per axis) moves 12 data bytes plus a couple of protocol/address bytes, call it 14 bytes total:

14 bytes × 22.5 µs/byte ≈ 315 µs

Plus a few microseconds of start/stop/repeated-start overhead — comfortably under the 0.4 ms line item budgeted above.

This is exactly the kind of arithmetic that separates "the sensor read is basically free" from "the sensor read might actually matter," and it only takes knowing the bus speed and the byte count to check.

Now watch what happens when inference — the single biggest line item, and the one most likely to grow as models get more capable — creeps from 6.0 ms to 8.0 ms, perhaps because someone swapped in a more accurate fusion model without re-checking the budget:

Sensor read                                    0.4 ms
Preprocessing                                  0.5 ms
Inference (grew: 6.0 ms → 8.0 ms)              8.0 ms
Actuation / output prep                        1.0 ms
Margin                                        -2.1 ms  (consumed entirely, then some)
--------------------------------------------------------
Total                                         11.9 ms   vs   10.0 ms budget

The loop now takes 11.9 ms to do work that's supposed to complete every 10 ms.

This isn't a "20% slower" problem in the way a data-center batch job being 20% slower is a throughput annoyance.

At a fixed 100 Hz sample rate, a cycle that takes 11.9 ms either steals time from the next cycle's sensor read — which means the next sample is read late, and the assumed fixed inter-sample interval the fusion filter's math depends on is now wrong — or the runtime has to drop the cycle entirely and skip that sample.

Either failure mode degrades the fusion estimate itself.

A Kalman filter or complementary filter derives its gains from an assumed, fixed dt between samples, and jittery or dropped samples inject exactly the kind of timing noise those filters aren't designed to absorb gracefully.

The system doesn't fail with an error message.

It fails by quietly getting worse at the one thing it exists to do — track orientation accurately.

This is why the margin line exists at all, and why it's a design decision, not slack you get to spend the moment you find it.

2.1 ms of margin at 100 Hz is protection against exactly this kind of creep: a model update, a firmware change to the I2C driver, a different actuation path — any of which can eat into a budget that looked comfortable the day it was designed.

A pipeline with zero margin is a pipeline where the very next change, of any kind, anywhere in the loop, is a latency-budget violation waiting to be discovered in the field rather than in a profiler.

A latency budget turns "the model is fast enough" into a question with a precise, checkable answer, rather than a feeling. Every stage gets an allocation. Every stage's measured time — not its estimated time, not its time on a bigger, un-throttled dev board — has to fit inside that allocation, simultaneously, every cycle, indefinitely. The moment one stage exceeds its allocation, the budget doesn't gracefully degrade — the whole cycle either misses its deadline outright or eats margin that was there to protect against exactly this.

5. Energy: The Budget That Doesn't Show Up in a Latency Profile

Everything so far has treated time as the scarce resource.

For a phone, a server, or a laptop, that's usually the right frame — there's a wall outlet or a large battery, and the practical constraint is almost always latency or throughput.

For a battery-powered wearable, that frame is incomplete in a way that can bite hard if it's not made explicit.

The device has to run for days between charges, off a battery that might store a few thousand joules total.

Two implementations can hit the exact same latency budget — both comfortably under 10 ms — and still differ by an order of magnitude in how long the device lasts before it needs to be plugged in.

Latency alone cannot see that difference.

Only energy can.

The foundational relationship, already introduced in the notes this lesson builds on, is:

E = P × t

E is energy in joules, P is power in watts, t is time in seconds.

It looks almost too simple to be a design tool, but the simplicity is exactly what makes it dangerous to skip: two implementations of the same function can trade P for t in either direction, and the one that "feels" faster isn't automatically the one that drains the battery slower.

Implementation A:  Power = 2.0 W,  Time = 10 ms   →  E = 2.0 × 0.010 = 0.020 J = 20 mJ
Implementation B:  Power = 1.0 W,  Time = 15 ms   →  E = 1.0 × 0.015 = 0.015 J = 15 mJ

Implementation A is 33% faster.

Implementation B uses 25% less energy per inference.

Neither number is wrong, and neither one alone tells you which implementation to ship — that depends entirely on which resource is actually scarce for the product.

If A blows the latency budget from Section 3, B wins regardless of energy.

If both fit comfortably inside the latency budget with margin to spare, B is very plausibly the better shipping choice for a battery-powered device.

The latency budget was already satisfied by both, at which point the leftover latency margin is worth nothing to the user — but the saved 5 mJ compounds every single cycle, all day, every day, for the life of the battery.

This is the core reframe this section exists to establish: on a battery-powered inference system, energy per inference — not latency — is very often the metric that actually determines the product's real-world usability, and it deserves to be measured and optimized as its own first-class quantity, not treated as a side effect that falls out automatically from optimizing latency.

6. Why Data Movement Costs More Than Compute

Energy per inference is the metric.

The next question is where that energy actually goes — and the answer turns out to be one of the most consequential, and most widely under-appreciated, facts in low-power computer engineering.

Moving a value costs far more energy than computing on it.

The canonical source for this comparison is Mark Horowitz's 2014 ISSCC keynote, "Computing's Energy Problem (and what we can do about it)," which tabulated the approximate energy cost, at a 45 nm process node, of individual arithmetic operations and memory accesses.

Those figures have since been reproduced constantly in the deep-learning-accelerator literature — most visibly in Sze, Chen, Yang, and Emer's widely cited survey Efficient Processing of Deep Neural Networks, which reuses Horowitz's numbers as the standard illustration of why accelerator design is dominated by data-movement concerns rather than raw arithmetic throughput.

The approximate figures, as commonly cited from that table:

Two things are worth flagging honestly about this table before drawing conclusions from it.

First, these are process-node-specific, illustrative figures from a single well-known source, not a universal physical constant.

A different fabrication node, a different memory technology, or a different measurement methodology will shift every number in the table, sometimes substantially.

Different secondary sources citing the same original Horowitz data don't even agree with each other to the last digit, which is itself a sign these are order-of-magnitude illustrations rather than a datasheet.

Second, precisely because of that variance, what should be trusted here is not any single pJ figure to two significant digits.

It's the ratio, which shows up consistently across every version of this comparison: a DRAM access costs roughly two orders of magnitude more energy than a basic arithmetic operation on the same data.

Taking the table above at face value, a 32-bit DRAM read (~640 pJ) costs about 200× a 32-bit floating-point multiply (~3.1 pJ), and close to four orders of magnitude more than an 8-bit integer add (~0.03 pJ).

Why should that be true at all, physically?

A logic gate flips a small, physically adjacent transistor's state — the capacitance being charged and discharged is tiny, because the wires are short and the whole circuit lives on a few square micrometers of silicon.

A DRAM access is a fundamentally different physical event.

It drives a signal down a comparatively long, capacitive on-chip interconnect — and, for off-chip DRAM specifically, an even longer, higher-capacitance PCB trace and package pin — activates an entire row of a memory array far larger than the datum actually being fetched, and pays a fixed row-activation and addressing overhead on top of the bits actually delivered.

None of that overhead exists for a value already sitting in a register or nearby SRAM.

The energy cost of moving a bit scales with the physical distance and capacitance it has to travel across, and DRAM is, by construction, physically far — logically adjacent to the compute in a diagram, but physically remote in silicon.

This is the same fact the Roofline model already taught, restated in a different unit. Lesson 1's Roofline post established that a kernel's latency is bounded by whichever is scarcer — compute throughput or memory bandwidth — and that arithmetic intensity (FLOPs per byte moved) determines which side of that ceiling a given kernel sits on. Energy obeys the identical shape of law: a kernel with low arithmetic intensity isn't just bandwidth-bound in time, it's data-movement-dominated in energy, for exactly the same underlying reason — the bottleneck resource, bytes crossing the boundary between DRAM and the compute core, is the same physical resource in both accountings. Roofline in seconds and roofline in joules are the same curve, traced against a different axis.

That reframe is what makes the rest of this series retroactively make sense as also being an energy story, not just a latency story.

Quantization (Lesson 3) doesn't just shrink compute — it shrinks bytes moved per value.

An int8 tensor moves a quarter of the bytes a float32 tensor does for the same number of elements, which under the table above is roughly a quarter of the DRAM-access energy for every read and write of that tensor, on top of whatever compute-energy savings int8 arithmetic itself provides.

Operator fusion (Lesson 2's graph optimization) avoids materializing an intermediate tensor to DRAM between two ops at all.

The value is produced, consumed, and discarded while still in a register or on-chip buffer.

Every intermediate tensor a fusion pass eliminates is a DRAM write and a DRAM read that simply never happen, at ~640 pJ each, times however many elements that tensor has.

Tiling and on-chip SRAM reuse (Lesson 4's kernel engineering) keep a working set resident in cache across many reuses instead of re-fetching it from DRAM on every access.

That's exactly the same mechanism the Roofline model uses to raise arithmetic intensity, now paying off in joules instead of just cycles saved.

Zero-copy memory planning (Lesson 5) avoids gratuitous buffer copies between pipeline stages.

Every copy avoided is again a DRAM round-trip, at DRAM's energy price, that the plan simply doesn't pay.

None of these techniques were introduced in this series as energy optimizations.

They were introduced as latency and throughput optimizations.

The Horowitz/Sze comparison is the reason they're also, almost automatically, energy optimizations: because the physical resource all four techniques economize on — bytes crossing the DRAM boundary — is precisely the resource that dominates both the time budget and the energy budget, for the same underlying physical reason.

7. DVFS: Trading Latency for Power, Explicitly

Section 6 was about what consumes energy — data movement versus compute.

This section is about a mechanism many embedded and mobile systems use to actively trade one resource for the other: Dynamic Voltage and Frequency Scaling (DVFS).

The relationship is a hardware-level lever on the same E = P × t equation from Section 5.

Running a core at higher clock frequency finishes work sooner — lower t — but drawing more instantaneous power to do it, since dynamic power in CMOS logic scales roughly with frequency and with the square of supply voltage, and higher frequency operation typically requires a higher supply voltage to switch reliably.

Running at lower frequency, and the correspondingly lower voltage it enables, draws less instantaneous power but takes longer to finish the same work.

Higher frequency  →  lower latency,  higher instantaneous power
Lower frequency   →  higher latency, lower instantaneous power

Because power scales roughly quadratically with voltage and voltage tracks frequency, dropping frequency (and voltage) by a given percentage tends to buy a larger percentage reduction in power than the percentage of latency given up.

That's exactly why DVFS is a net energy win in many workloads, not just a wash.

An inference runtime with DVFS control has to decide, per stage or even per kernel, which operating point to run at — and two competing strategies are worth naming explicitly, because intuition alone doesn't reliably pick the winner.

Race-to-idle runs at the highest available frequency to finish the work as fast as possible, then drops the core into a deep low-power idle state for whatever time remains in the cycle.

This wins when the idle-state power draw is dramatically lower than active power — roughly, active power well above idle power — and the overhead of entering and leaving that idle state is small relative to the time saved.

Many modern SoCs are built with exactly this asymmetry, because idle states can gate clocks and power rails almost entirely.

Run-slow-and-steady picks a lower, sustained frequency that still finishes inside the latency budget, without ever fully idling.

This can win instead when active power doesn't scale down as favorably as expected at high frequency — some designs pay disproportionate power for the last bit of clock headroom — when idle-state transitions carry real energy or latency overhead of their own, or when other subsystems (a radio, a display, always-on peripherals) keep drawing a floor of power regardless of what the compute core is doing, which erodes the advantage of finishing early if there's nothing to shut off.

The Implementation A/B numbers from Section 5 are exactly this tradeoff, worked concretely.

A races to a finish in 10 ms at 2 W (20 mJ).

B stretches the same work over 15 ms at 1 W (15 mJ).

Under this section's framing, A is closer to a race-to-idle strategy and B closer to run-slow-and-steady — and B wins the energy comparison here specifically because the power reduction (2 W → 1 W, a 2× cut) outpaces the latency increase (10 ms → 15 ms, a 1.5× increase).

Whether that holds for a real chip is an empirical question about that chip's specific DVFS curve and idle-state power, not something to assume from the shape of the argument alone.

That's exactly why this is a measurement problem, not a rule of thumb to apply blindly.

DVFS is proof that "faster" and "lower energy" are not the same optimization target, and sometimes point in opposite directions. A runtime tuned purely for lowest latency will race to idle every time. A runtime tuned for battery life needs to actually measure both ends of the frequency/voltage curve for its specific silicon before picking an operating point — the right answer depends on idle power, transition overhead, and what else on the board is drawing power regardless, none of which can be read off a clock-speed number alone.

8. A Joules-Per-Inference Estimate, Worked

Put Sections 6 and 7's ideas together into one end-to-end estimate, in the spirit of the arithmetic Section 2 did for latency.

This is explicitly an illustrative, order-of-magnitude estimate built from the approximate per-operation figures in Section 6's table — not a measurement of any specific real chip.

It's presented that way deliberately, to show the shape of where energy goes rather than to claim a precise number for any particular piece of silicon.

Take a small quantized int8 model doing 5 million MAC operations per inference, with 200 KB of weights — too large to fit in a small MCU's on-chip SRAM (say, a 64 KB budget), so the weights have to be streamed from DRAM once per inference, worst case, with no cross-inference weight caching.

Compute energy.

Approximating an int8 MAC's cost from the add/multiply figures in Section 6's table — an 8-bit multiply plausibly costing somewhere in the same rough neighborhood as the cited 16-bit floating-point multiply once accumulation overhead is included, so call it roughly ~0.2 pJ per MAC as an illustrative round number, clearly flagged as an approximation rather than an independently verified figure for int8 specifically:

5,000,000 MACs × ~0.2 pJ/MAC ≈ 1,000,000 pJ = 1 µJ

Data-movement energy (worst case — DRAM every time).

200 KB is 51,200 32-bit words.

At the cited ~640 pJ per 32-bit DRAM read:

51,200 words × 640 pJ ≈ 32,768,000 pJ ≈ 32.8 µJ

Total, worst case:

Compute:        1.0 µJ   (~3%)
Weight fetch:   32.8 µJ  (~97%)
------------------------------
Total:         ~33.8 µJ per inference

Ninety-seven percent of the energy in this illustrative estimate goes to moving weights from DRAM, not to the arithmetic those weights are used for.

That's precisely Section 6's claim, made concrete with numbers.

As a sanity check against real measured systems rather than this back-of-envelope model: a recent study deploying an 8-bit-quantized TinyML object-detection model on real embedded hardware measured 10.6–22.1 µJ per inference end-to-end.

That's the same order of magnitude as this illustrative estimate — close enough to trust the shape of the argument, while still being a genuinely different, independently measured system, not a number this estimate was tuned to match.

Now the contrast that makes the whole section worth doing.

What if those same weights fit in on-chip SRAM instead — via more aggressive quantization, a smaller model, or tiling that guarantees each weight is loaded once and reused, rather than re-streamed from DRAM every inference?

Read cost drops from ~640 pJ per 32-bit word to roughly ~5 pJ:

51,200 words × 5 pJ ≈ 256,000 pJ ≈ 0.26 µJ
 
Compute:        1.0 µJ
Weight fetch:   0.26 µJ
------------------------------
Total:         ~1.26 µJ per inference   (≈27× less than the DRAM case)

That 27× gap, in this illustrative model, is entirely attributable to where the weights physically live.

Nothing about the arithmetic changed at all.

Scaled up to a continuous 100 Hz duty cycle, the difference stops being an abstract ratio and becomes a felt product outcome.

At ~33.8 µJ/inference, 100 Hz continuous operation draws roughly 33.8 µJ × 100 ≈ 3.38 mW of average power just for this component.

Against a representative small wearable battery — 150 mAh at 3.7 V, or about 0.150 Ah × 3.7 V × 3600 s ≈ 1998 J of stored energy — 3.38 mW alone would exhaust that entire budget in roughly 1998 J / 0.00338 W ≈ 591,000 s, about 6.8 days, before counting the sensor, the radio, the display, or anything else on the board.

At ~1.26 µJ/inference, with weights resident in SRAM, the same duty cycle draws ~126 µW, and the same battery budget stretches to roughly 1998 J / 0.000126 W ≈ 15.9 million seconds, about 184 days — six months, for the identical inference workload, differing only in where the weights live during the read.

That gap is not a rounding error to optimize later.

It is very plausibly the difference between a product that needs charging nightly and one that needs charging monthly, and the entire cause of the difference is a memory-placement decision, not a change to a single FLOP of arithmetic.

9. The Throughline

Every technique this lesson touches — Amdahl-guided prioritization, latency budgeting, quantization, fusion, tiling, zero-copy planning, DVFS — is downstream of one underlying fact, restated once more because it's worth carrying forward explicitly.

On real hardware, moving data costs more than computing on it, in both time and energy, for the same physical reason in both cases.

The Roofline model made that fact legible in seconds.

This lesson made it legible in joules.

They are not two separate lessons that happen to rhyme — they are one lesson, measured on two different axes, and every optimization from earlier in this series that reduced bytes moved was quietly paying down both bills at once, whether or not it was framed that way at the time.

For a battery-powered, duty-cycled system — precisely the class of hardware a health wearable's sensor-fusion and gesture-recognition pipelines belong to — that equivalence stops being a tidy observation and becomes the actual design constraint.

The latency budget from Section 4 determines whether the product works at all, cycle to cycle.

The energy budget from Sections 5 through 8 determines whether the product is usable — whether it survives a day, a week, or a month between charges.

Both budgets have to be satisfied simultaneously, by the same pipeline, and neither one is optional.

Further Reading

• Amdahl, G. M., "Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities," AFIPS Spring Joint Computer Conference, 1967 — the original paper this entire lesson's first section builds on. PDF via Stony Brook CSE613 course archive

• Horowitz, M., "1.1 Computing's Energy Problem (and what we can do about it)," ISSCC 2014 keynote — the source of the widely cited per-operation and per-access energy figures used throughout Section 6, reproduced here via a public mirror of the presentation. gwern.net PDF mirror

• Sze, V., Chen, Y-H., Yang, T-J., and Emer, J. S., Efficient Processing of Deep Neural Networks: A Tutorial and Survey — the survey that popularized Horowitz's energy table as the standard illustration for why DNN accelerator design is dominated by data-movement concerns. Excerpt via MIT Eyeriss project

• GeeksforGeeks, "Amdahl's Law and its Proof" — a compact derivation and worked examples of the general form of the law used in Section 1. geeksforgeeks.org

• "Deploying TinyML for energy-efficient object detection and communication in low-power edge AI systems," Scientific Reports, 2025 — the source of the measured 10.6–22.1 µJ per-inference figure used as a sanity check in Section 8. nature.com

• "Dynamic Voltage and Frequency Scaling as a Method for Reducing Energy Consumption in Ultra-Low-Power Embedded Systems," Electronics, MDPI, 2024 — a focused treatment of DVFS specifically in the ultra-low-power embedded regime discussed in Section 7. mdpi.com

Part 2 of this lesson, "Embedded and Microcontroller Inference," picks up exactly where the energy discussion here leaves off — the specific constraints of running on hardware with kilobytes of SRAM and no DRAM at all, and what that does to every technique this series has built so far.