What it means
Throughput measures how many tokens a model can push through per second, usually aggregated across all the requests packed into a batch. A related idea, goodput, counts only the throughput that actually meets latency targets, since serving too many users at once can slow each one below an acceptable response time. Throughput is largely governed by memory bandwidth and by how efficiently requests are batched, not just by raw compute. In the economics of the AI supply chain it is decisive: higher throughput spreads fixed hardware cost across more tokens, directly lowering cost per token. Because inference is the recurring workload that pays for the stack beneath it, squeezing more throughput out of each accelerator is one of the strongest levers on serving margins.
Why it matters to investors
Throughput sets how cheaply a provider can serve tokens, so gains here flow straight into lower cost per token and better inference margins. Serving-focused players like Together AI, Microsoft and Alphabet optimize it aggressively because inference is the workload that monetizes the whole stack.
Companies on this part of the chain
Named to show where the term sits in the AI supply chain — research, not advice, and never a recommendation to buy or sell.
Related terms
See Throughput in the live AI chain.
THE ENTITY maps every constraint onto one live model — which part is tight now, who owns it, and who gets squeezed when it moves. Plain-English reads you can check.
THE ENTITY is an educational read on the AI supply chain — research, not investment advice. It explains how the chain works and who sits where, never price targets or buy/sell calls.