Audio editionListen to this book10 chapters · 30 min
01
Before testing a claim it is worth knowing who is making it, and which of its two incompatible meanings is on the table.
The phrase has a provenance, and it is not an analyst's. In the launch post for an open inference benchmark, AMD's vice president of GPU software is quoted saying exactly four words: "Speed is the moat."c-023 He is describing his own company's software stack, on a benchmark his company participates in, at a moment when its principal competitor holds the performance narrative.
Read the same document a few paragraphs up and Nvidia's chief executive describes the identical benchmark results in an entirely different vocabulary — performance per dollar and per megawatt.c-029 Two vendors, one dataset, two framings. The one behind in raw throughput talks about efficiency; the one with a fast software story talks about speed. That is positioning rather than physics, and it is the first reason to hold the slogan at arm's length.
Speed is the moat. Anush Elangovan, VP GPU Software, AMD — quoted in the InferenceMAX launch postc-023
None of which makes it wrong. Interested parties are often correct, and a claim's origin is a reason to test it carefully rather than a reason to dismiss it. But it does explain why the phrase travels so well: it is a good line, and somebody with a budget wanted it to travel.
Two claims wearing one sentence
The phrase carries two meanings that are routinely conflated, and they are not the same argument.
The first is about organizational velocity: ship fast, learn fast, compound the advantage. That is a claim about how a company operates, and it is genuinely interesting. It is also not what this book examines. It was ruled out of scope in writing before any evidence was gathered, precisely because the two meanings are so easy to slide between.
The second is narrower and measurable: that inference speed — latency and throughput, the rate at which a model returns tokens — is a durable differentiator for the systems built on it. That claim can be tested against benchmarks, price lists, contracts, and controlled experiments. It is the one tested here.
What the word has to earn
A moat is not an advantage. It is an advantage a competitor cannot cross. Faster is a fact about today; a moat is a claim about how long today lasts. Applied to inference, the word implies three things: that a meaningful gap exists between the fast and the slow, that the gap persists rather than closing, and that the gap produces something a business can actually bank.
The chapters that follow take those in order — the size of the gap, its durability, and whether it converts. The first survives contact with the evidence. The second does not. The third turns out to depend entirely on what kind of work you are running, and the received wisdom about it is wrong in a direction almost nobody expects.
02
The thesis assumes token economics are a solved problem and speed is the successor axis. Half of that holds.
There is no serious dispute about the direction. Epoch AI measured the price of reaching a fixed capability milestone across six benchmarks over three years and found it falling between nine and nine hundred times per year, with a median of fifty.c-001 Restricting the data to models released after January 2024 raises that median to two hundred.c-002 Whatever else is true of this market, things are getting cheaper at a rate with no obvious precedent.
The magnitude, though, is contested in a way worth understanding. Andreessen Horowitz puts the decline at ten times per year for a model of equivalent performancec-004 — five times slower than Epoch's median. Neither number is wrong. They are measuring different objects: Epoch fits a regression across capability milestones on six benchmarks, while the venture firm tracks a single equivalence curve. Epoch also cautions that its fastest observed rates are recent and may not hold.c-003
A five-fold disagreement between two credible sources on the same phenomenon, resolved entirely by what each chose to count. This recurs throughout the subject: "tokens per second" means output-only or total, single-stream or aggregate, depending on who is publishing. Comparing figures built on different definitions is the most common way to be precisely wrong here.
Understood is not finished
Managing AI spend has become near-universal practice. Ninety-eight percent of 1,192 respondents to the sixth annual State of FinOps survey now manage it, up from thirty-one percent two years earlier.c-040 On that evidence the thesis is right that cost is understood.
It does not follow that cost has stopped being the competitive axis. In July 2026 OpenAI cut two GPT-5.6 tiers by twenty and eighty percent respectivelyc-005 — roughly three weeks after those models were released — and its own published price list confirms the cheapest tier at twenty cents and one dollar twenty per million tokens in and out.c-080 Contemporary reporting attributes the move to pressure from a cost-sensitive customer base.c-007 The company framed it as capability and efficiency advancing together.c-006
A market where a leading vendor cuts prices eighty percent three weeks after launch is not a market that has finished competing on price.c-100
The two axes are one curve
The deeper problem with the sequential framing is mechanical. Latency and throughput are not independent of cost; they are the same dial viewed from different ends.
Serving works by batching. Pack more requests onto a GPU and each one costs less to serve, because fixed capacity is amortized across more work — but every request in the batch waits longer, sharing attention with its neighbors. Small batches give low latency and poor economics; large batches invert both. On fixed hardware you cannot have minimum latency and maximum throughput at once.c-025 A second benchmark, run independently, reproduces the same structure and insists on reporting performance as a curve rather than a number.c-072
Where a provider sets their batch size is a business decision as much as a technical one. On the economics of inference servingc-026
Which reframes the whole question. A speed program run on the assumption that cost is finished will keep rediscovering that its latency wins arrive as cost increases, and that the cheap tier it was told to use is slow for a reason. These are not two axes to be tackled in sequence. They are one surface, and the only real decision is where to sit on it.
03
A moat needs a gap. The gap is real, larger than most people assume, and it disappears entirely under one common condition.
Take a single open-weight model and ask what happens when different companies serve it. Gemma 4 31B, at a 131,000-token context: Cerebras returns a median 1,357 output tokens per second and completes in 2.43 seconds.c-010 SambaNova, serving the identical model at the identical context length, returns 201 tokens per second and takes 13.49 seconds.c-011
Same weights. Same context. Roughly seven times the generation rate and five and a half times the wall clock. And the slower endpoint is the cheaper one — ten cents per task against twenty-four.c-011
The pattern repeats on a different model. Qwen3.6 27B through Groq returns 463 tokens per second and finishes in 14.58 secondsc-012; through DeepInfra, 53 tokens per second and 117.94 seconds.c-013 That pair deserves an asterisk — the slower deployment is an FP8 quantization at a longer 262k context, so the comparison is not like-for-like.c-013 The Gemma pair, matched on model and context, is the cleaner evidence.
Then the finding that reorganizes the question
Now run the same exercise on a proprietary frontier model, where the choice is not which company serves the weights but which storefront sells them. Claude Opus 5 through Amazon Bedrock: 44.88 seconds.c-014 Through Anthropic's own API: 54.58 seconds.c-018 Through Google: 61.68 seconds.c-019
About 1.4 times, end to end. The gap has all but closed.
The threshold set in advance was two times — below that, a speed advantage is not worth organizing around. Open weights clear it three to four times over.c-101 Proprietary models fall under it.c-102 So the size of the prize turns on a structural question rather than an engineering one: can you choose who serves the weights? If your system is built on a frontier proprietary model, there is very little speed available to shop for, and the decision is close to moot.
Every cross-provider figure above comes from a single point-in-time snapshot of one benchmarker. It publishes its method, its cadence, and its test location, which is why it is used — but it is one source. A second benchmark corroborates that large dispersion exists and that performance is a curve,c-070c-072 yet it measures hardware and serving stacks rather than API endpoints. It does not independently confirm that Cerebras is 6.8 times SambaNova on this model.
Three measurement caveats survive into everything above. Output speed is defined as tokens per second after the first token, so the headline rate excludes the wait a user actually feels.c-015 Figures are medians over a trailing 72 hours rather than single readings.c-016 And time-to-first-token includes network latency from one test location, which the publisher acknowledges may advantage or disadvantage providers by geography.c-017
04
A gap is necessary but not sufficient. The question is whether anyone can hold one, and here the thesis breaks cleanly.
Consider what used to distinguish a fast serving stack. Paged attention for KV cache memory. Continuous batching. Prefix caching. Speculative decoding. Disaggregated prefill and decode. Each was, at introduction, a genuine engineering advantage held by whoever implemented it first.
All of them now appear in a single line of the open-source vLLM documentation, under the heading of advanced features.c-024 Not as a roadmap. As what the software does when you install it.
The frontier moves faster than the reader
The rate of change is documented by the people who measure it for a living. Inference software improves through kernel-level optimization, distributed serving strategies, and scheduling work that lands "in incremental releases that can be just days apart."c-020 The consequence, stated plainly by the same benchmarker: results taken at a fixed point in time go stale and stop representing what the stack can actually do.c-021
The industry's response is instructive. Not better benchmarks — continuous ones. The suite is re-run nightly across hundreds of chips.c-022 When an entire measurement discipline reorganizes itself around the assumption that last month's numbers are fiction, it is describing a field where advantages do not sit still.
Putting a number on it
Characterization is not measurement, and for a while this was the weakest part of the argument. It is no longer. A second benchmark, run independently of the first, records AMD's inference software improving by up to two times between December 2025 and January 2026.c-070 Over a slightly longer window it recorded throughput at equal interactivity nearly doubling in under two months.c-071
On unchanged silicon. The same chips, in the same racks, delivering twice the work because the software around them got better.
The half-life of a software-derived speed lead is roughly one to two months. Most procurement cycles are longer than that. The central findingc-112
This is the number that decides the question. An advantage that halves in weeks cannot be defended on any horizon on which the word "moat" is normally used. By the time a vendor evaluation is written, circulated, approved, and signed, the thing it measured has moved.c-103
What does survive
Two candidates, both resting on thinner evidence than the finding above, and both worth stating because they are the strongest case against it.
Custom inference silicon reaches a low-latency tier that GPU-based providers cannot match by tuning batch size alone — while occupying what the same source calls a distinct and expensive corner of the market.c-028 Whether that corner stays expensive is exactly what would decide the question, and this corpus cannot settle it: the only source making the argument is the source qualifying it.
Separately, model labs hold a structural cost advantage in serving, because they can backfill idle capacity with training runs, research ablations, and offline batch work. The fleet is never truly idle, and its cost amortizes across everything.c-027 That is a real durable edge. It is an edge in economics rather than in speed.
Which points at the reframing this chapter argues for. What persists is not pace but position — where an operator chooses to sit on the latency-throughput curve, a choice made commercially and revisable at any time.c-104
05
The largest wall-clock difference available for a fixed budget is not an engineering achievement. It is a checkbox.
Anyone who has watched a batch job crawl has an intuition that the same work, at the same budget, can take a day or a month depending on how it is run. The intuition is sound in structure. It is also, in the documented case, a product decision rather than a capability.
Anthropic's Message Batches API charges fifty percent of standard prices, with most batches finishing in under an hour.c-030 Results become available when everything completes or after twenty-four hours, whichever comes first, and anything unfinished at that point expires.c-031 OpenAI's batch offering is the same trade in the same shape: half the cost, higher rate limits, a twenty-four hour turnaround.c-032 Even inside that window the work is not instant — a single 300,000-token generation can take over an hour.c-033
| Tier | Price | Latency commitment | Wall clock |
|---|---|---|---|
| Priority | premium | defined target per model | seconds |
| Standard | list | none | seconds |
| Batch | 50% | none | ≤ 24h |
So: identical work, identical model, half the budget, and a wall clock that moves from seconds to a day.c-105 The structure of the folk claim is confirmed. Its magnitude is not — the documented ceiling is twenty-four hours per batch,c-031c-032 and nothing found here supports the month-long figure that often accompanies the argument.
Latency is also sold directly. Priority requests are prioritized over all others, which is the whole product.c-034
A counter-signal worth sitting with
If buying latency were a growing market, vendors would be expanding the mechanism for selling it. At least one is doing the opposite: Anthropic states that Priority Tier capacity commitments are no longer available for purchase.c-035 A withdrawn product is not proof of weak demand — capacity constraints would produce the same notice — but it is not the move a growing market predicts either.
Where the compounding actually happens
Single-turn chat was forgiving of a slow model, because one exchange means one wait. Agentic work removes that forgiveness by multiplying the number of sequential calls per task.
Production traces bear this out. An office-work trace averages about forty-one turns.c-036 Code-QA traces run widest of all, with tool-call turns reaching two hundred.c-039 And the distributions are heavy-tailed at every dimension that matters — number of turns, tokens generated per turn, tool output size, tool call latency.c-037
Multiply a per-token rate by two hundred sequential round trips and the difference between fast and slow serving stops being a user-experience detail and becomes the difference between a task that finishes inside a working day and one that does not.c-108 If there is a case for treating speed as strategic, this is where it lives — not in the interface, but in the loop.
06
The business case for low latency rests on a number from a different decade and a different interaction. Someone finally tested the real thing.
Ask why response speed matters and the answer arrives with remarkable consistency: Amazon found that every hundred milliseconds of delay cost one percent in sales. The figure appears in vendor decks, engineering blogs, and product briefs, usually without a date and never with a method.c-038
It is a real finding about e-commerce page loads. It is not a finding about language model inference, and the distance between those two things is not small: a page either renders or it does not, whereas a model streams a response whose content the reader is actively consuming while it arrives.c-106 Citing it in an inference context is citation laundering — a number that everyone repeats and nobody recomputed.
A controlled test, at last
A study published at the 2026 CHI conference ran the experiment directly. Two hundred and forty participants, randomly assigned, in a three-by-two between-subjects design: time-to-first-token fixed at 2, 9, or 20 seconds, across two categories of knowledge work.c-050 Every participant completed three tasks at their assigned latency, with system logs capturing behavior and surveys capturing perception.
The behavioral result was flat. Prompting rates, copying, pasting, refreshing — none of it shifted with latency. What did drive behavior was the kind of task being done.c-051 Across a tenfold range in response delay, users did not measurably change how they worked.
The perception result ran backwards.
Participants who waited two seconds rated the model's output less thoughtful and less useful than participants who waited nine to twenty. Controlled experiment, n=240, CHI 2026c-052
Delay, in this setting, read as deliberation. Speed read as carelessness. The authors' conclusion is that latency is not simply a cost to be minimized but a tunable design variablec-053 — a claim that would have sounded perverse in a product review a year ago.
One experiment, one band of latency, one family of tasks, crowdsourced participants. It says nothing about voice agents, trading systems, or anything with a hard real-time constraint, and nothing about the thirty-second-plus waits that long-horizon agentic work produces.c-050
It is also the only controlled study in this corpus, and it is better evidence than what it displaces. Within the range tested, the interactive case for speed is not merely unproven — it is contradicted by the best available measurement.c-110 A second study finding the opposite would restore the conventional view, and would be worth running.
Taken with the previous chapter, the shape of the answer emerges. The throughput case for speed is well evidenced and mechanically obvious. The interactive case — the one that justifies most latency engineering budgets — is resting on a borrowed statistic, and the one time somebody checked, it did not hold.
07
The premise says leaders have not yet made speed a strategic axis. The contracts say otherwise.
Financial governance of AI has matured quickly. Ninety-eight percent of State of FinOps respondents now manage AI spend, and the priorities that displaced pure cost optimization are governance, forecasting, organizational alignment, and expanding coverage.c-041 Mature practice is described as moving toward unit economics and influencing technology selectionc-042 — precisely the surface a speed criterion would have to occupy.
Latency does not appear in that list. It would be easy, and wrong, to read that as proof that nobody governs speed. Absence of mention in one summary of one survey is not a measurement, and treating it as one is how a plausible story gets mistaken for a finding.
Read the contract instead
The direct evidence is in the deployment documentation, where it has been sitting in plain sight. Microsoft's table of deployment types carries a column headed Latency Service Level Agreement. Priority processing carries a defined latency target per model. Provisioned throughput carries a defined latency target per model.c-060 Standard deployments carry none. Batch carries none.c-061
| Deployment | Billing | Latency SLA |
|---|---|---|
| Standard | per token | None |
| Priority processing | per token, priority rate | Defined target per model |
| Provisioned | per PTU per hour | Defined target per model |
| Batch | discounted per token | None |
Speed is not an unclaimed strategic axis awaiting an owner. It is a priced, tiered, contractually specified product, and what a buyer gives up by taking the cheap tier is exactly the latency guarantee.c-111
That changes what the problem is. The gap is not strategic imagination. It is that most organizations are sitting on the ungoverned tier by default rather than by decision — which is a procurement failure, and a considerably easier one to fix than a research program.
The measurement discipline agrees
The same institutionalization is visible where systems get benchmarked. MLPerf Inference v5.1 expanded an interactive scenario that tests performance under tighter latency constraints, motivated explicitly by agentic applications.c-043 That round drew a record twenty-seven submitting organizations.c-044 Latency is being standardized where vendors are measured, whatever is happening inside any given enterprise.c-107
What the contracts establish is that platforms sell latency. Whether buying organizations set and enforce their own internal latency objectives, and hold vendors to those targets, would require procurement documents this project did not obtain. That question remains genuinely unanswered.
08
The finding is not that speed is unimportant. It is that almost everyone is buying it in the wrong place.
Speed deserves an owner. Not the one the slogan implies — a team hunting the fastest provider, on a chase with no end, against a lead that halves in weeks.c-112 The job worth staffing owns placement: which workloads sit on which contractual tier, given that the tiers already exist and already price latency.c-060
That is a procurement function with an engineering input, and it is unglamorous in a way that probably explains why it goes unstaffed.
Classify workloads by tier before comparing vendors. The batch, standard, and priority split holds the largest wall-clock difference available for a fixed budgetc-030c-032 and requires no change of provider. Most organizations run everything on one tier because nobody was asked to decide.
Treat open-weight versus proprietary as the gate. Shopping for speed pays five to eight times on open weightsc-101 and close to nothing on proprietary models.c-102 Model-class choice determines whether a speed program has anything to work with at all — settle that before funding the program.
Buy the latency you need and stop paying for the rest. A latency target is a contract term available on requestc-060; on the standard tier there is nothing to enforce.c-061 Both halves of that sentence cost money in opposite directions.
Do not assume faster is better in the interface. Within two to twenty seconds on knowledge tasks, faster changed no behavior and read as less thoughtful.c-052 Spend the latency budget where it compounds — agentic loops running dozens to hundreds of sequential turnsc-036c-108 — rather than on shaving an interactive response users may not reward.
The thesis, restated honestly
"Speed is the moat" fails on the evidence gathered here. Serving optimizations diffuse into open-source defaultsc-024, the frontier advances in daysc-020, a measured lead halves inside two monthsc-112, position on the curve is purchased rather than defendedc-104, and the interactive business case is contradicted by the only controlled study available.c-110
What survives is narrower and more useful. Dispersion between providers is real and large where weights are openly served.c-101 It persists as a feature of the market rather than closing, because it originates in where operators choose to sit on a curve rather than in anyone's cleverness. And agentic work multiplies per-call latency into wall-clock outcomes in a way single-turn chat never did.c-108
So the durable asset is not being fast. It is being organized to keep choosing — to re-examine placement on the cost, latency, and throughput surface more often than a frontier moving in days will tolerate. That is an advantage built from attention and process rather than from technology, which makes it unglamorous, cheap to start, and available to anyone willing to treat it as somebody's actual job.
How this was assembled
The work ran through a fixed pipeline: scope, gather, verify, synthesize, report. The ordering is the method. The question, its sub-questions, the out-of-scope list, and the evidence that would settle the matter were written before any source was read. Sources were then captured to disk and hashed; a source that was not snapshotted does not exist for this project. Each claim was bound to an exact quotation inside a captured file, and a verification gate checked every binding before the report was allowed to ship.
The gate fails a document when a citation marker resolves to nothing, when a quoted passage does not occur in its snapshot, when a snapshot's hash has changed since capture, or when a contested claim is presented as settled. It returned PASS with zero hard failures across 65 claims and 33 sources.
Source tiers
Every source carries a tier, from T1 to T4. The tier answers one question and only one: how directly can this be checked? It is not a rating of quality, and it says nothing about whether the source agreed with the argument. A vendor benchmark that supported the conclusion still scored T4; a price list that undercut it still scored T1.
The scale used here is the technical one, adjusted for the pricing and governance questions, where the evidence that settles a point is commercial rather than engineering. The full definitions sit beside the ledger in Appendix B. One rule cut across all four: any latency or throughput figure older than twelve months was treated as a historical trend point, never as a current one.
Two passes, and two reversals
A first pass cleared the gate with six declared gaps. A second closed four of them, and two of those closures changed the answer rather than supporting it. The question of whether latency affects user behavior moved from "no independent evidence exists" to "a controlled experiment exists and it points the other way." The question of whether organizations govern speed moved from "unresolved" to "answered, against the thesis." Both reversals are recorded in the chapters where they occurred.
What this does not establish
What would change the conclusion
The verdict turns toward "moat" if the custom-silicon tier proves unreachable by batch tuning across several model generations rather than being a temporary and expensive corner.c-028 It turns further away if the open-weight dispersion proves to be an artifact of one benchmarker's test geography or prompt mix.c-017 The finding most exposed to better evidence is the interactive-latency reversal, which is a single study.c-052 The finding least exposed is the erosion rate: two independent benchmarks, measuring different layers, both report software gains arriving faster than procurement can respond.c-112
Thirty-three sources, each saved and fingerprinted at the moment it was read. The claim identifiers throughout this book resolve to entries in the ledger at tech.anujsadani.in/speed-is-the-moat, which names the source behind every claim and quotes the words it rests on. Tier reflects how directly a source can be checked, never whether its content was welcome.
What the tiers mean
| ID | Tier | Publisher | Title |
|---|---|---|---|
| s-001 | T2 | Epoch AI | LLM inference prices have fallen rapidly but unequally across tasks |
| s-002 | T2 | Artificial Analysis | Language Model API Performance Benchmarking (methodology) |
| s-003 | T4 | Andreessen Horowitz | Welcome to LLMflation |
| s-004 | T3 | mlechner | The Economics of LLM Inference: Batch Sizes, Latency Tiers |
| s-005 | T2 | MLCommons | MLPerf Inference v5.1 benchmark results |
| s-006 | T2 | vLLM project | Inside vLLM: anatomy of a high-throughput inference system |
| s-007 | T3 | Inclusion AI | The community stories of vLLM and SGLang |
| s-008 | T3 | Applied Compute | Benchmarking inference engines on agentic workloads |
| s-009 | T3 | FinOps Weekly | AI cost management and FinOps: from blind spots to governance |
| s-010 | T4 | GMI Cloud | Fastest open model inference — capture failed, unused |
| s-011 | T2 | arXiv | The shift to agentic AI: evidence from Codex |
| s-012 | T1 | OpenAI | Batch API guide |
| s-013 | T1 | Anthropic | Batch processing — Claude docs |
| s-014 | T2 | Artificial Analysis | LLM API provider leaderboard |
| s-015 | T4 | Dataiku | Seven best LLM tools for businesses in 2026 |
| s-016 | T3 | CNBC | OpenAI cuts prices for two of its GPT-5.6 models |
| s-017 | T4 | Alhena AI | E-commerce latency: 100ms = 1% revenue lost |
| s-018 | T2 | FinOps Foundation | State of FinOps survey 2026 |
| s-019 | T3 | Constellation Research | Nvidia GTC 2026: hardware strategy beyond the GPU |
| s-020 | T4 | Tech Times | Custom AI chips outpace Nvidia GPU growth in 2026 |
| s-021 | T1 | Anthropic | Service tiers — Claude API docs |
| s-022 | T2 | SemiAnalysis | InferenceMAX: open source inference benchmarking |
| s-023 | T4 | Together AI | Fastest inference for the top open-source models |
| s-024 | T2 | CHI 2026 | The impact of response latency and task type on human-LLM interaction and perception |
| s-025 | T1 | Microsoft | Azure OpenAI performance and latency |
| s-026 | T1 | SOSP 2023 | Efficient memory management for LLM serving with PagedAttention |
| s-027 | T1 | Microsoft | Azure OpenAI provisioned throughput concepts |
| s-028 | T2 | SemiAnalysis | InferenceX v2: Blackwell vs AMD vs Hopper |
| s-029 | T3 | Tom's Hardware | InferenceMax tests software stacks, efficiency, and TCO |
| s-030 | T1 | OpenAI | API pricing — HTML route, navigation only |
| s-031 | T1 | vLLM project | Release history |
| s-032 | T1 | OpenAI | API pricing (markdown route) |
| s-033 | T4 | GMI Cloud | Fastest open model inference (re-capture of s-010) |
Anuj Sadani builds AI systems and the teams that build them. He spent a decade at NVIDIA, has worked inside AI-first organizations, and has led multicultural engineering pods across Europe — sixteen-plus years, most of them on the AI era's unglamorous half: turning hype into systems that ship, and systems that ship into outcomes that hold up in production. That work has fed into industry recognition from Gartner and Everest Group, and an Innovator of the Year nod.
He is the author of The Clean Vibe Coder: A Code of Conduct for Programmers in the Age of AI Agents and Borrow the Line. Own the Move., and writes about engineering, leadership, and what stays human when the tools get good. He still believes the best technology is the kind that makes the people around it braver.
This book came out of a working habit rather than a thesis: checking whether the sentences everyone repeats survive contact with the evidence. Most of the time they half survive, and the half that does is the useful part.
anujsadani.in
tech.anujsadani.in