Skip to content
AI supply chain term

KV cache

A KV cache is the stored key and value tensors from earlier tokens that let a language model generate each new token without recomputing the whole sequence.

What it means

The KV cache is a memory optimization inside transformer inference. As a model generates text, each new token must attend to every token that came before it. Rather than recomputing the key and value projections for the entire history at every step, the model saves them in a key-value cache and reuses them. This turns a repeated, quadratic-cost operation into an efficient lookup. The catch is that the cache grows with context length and with the number of concurrent requests, so it consumes scarce high-bandwidth memory capacity and bandwidth. In inference serving, the KV cache is often the binding constraint: it caps how long a context can be and how many users a single accelerator can serve at once, which is why memory and techniques like quantization matter so much.

Why it matters to investors

The KV cache makes inference serving memory-bound rather than compute-bound, so it drives demand for high-bandwidth memory and shapes how cheaply providers can serve long-context and high-concurrency workloads. Firms running large-scale inference such as OpenAI, Microsoft and Alphabet manage cache size directly because it governs serving cost and capacity.

Companies on this part of the chain

OpenAIMicrosoftMSFTAlphabet (Google)GOOGLTogether AI

Named to show where the term sits in the AI supply chain — research, not advice, and never a recommendation to buy or sell.

Related terms

Context windowA context window is the maximum amount of text, measured in tokens, that a model can consider at one time.HBM (High-Bandwidth Memory)High-Bandwidth Memory — DRAM stacked next to an accelerator to feed it data fast; often the real limit on GPU output.InferenceRunning a trained model to answer requests — the recurring cost of AI in production.Throughput (tokens/sec)Throughput is the number of tokens an AI system can generate or process per second, a core measure of inference serving efficiency.

See KV cache in the live AI chain.

THE ENTITY maps every constraint onto one live model — which part is tight now, who owns it, and who gets squeezed when it moves. Plain-English reads you can check.

Explore the features
All glossary terms

THE ENTITY is an educational read on the AI supply chain — research, not investment advice. It explains how the chain works and who sits where, never price targets or buy/sell calls.