NVIDIA Argues 3 Percent More HBM Is Enough: Up to 30x Agent Throughput per Dollar via Full-Stack Codesign

NVIDIA published a technical blog on Thursday laying out its answer to a problem the industry has been circling for months: GPU memory, not compute, is becoming the binding constraint on AI infrastructure. The company's argument is that the era of solving memory pressure by adding more HBM is ending, and that full-stack co-design of GPUs, CPUs, networking, storage and software is where the next gains live.

The pressure comes from three directions at once. Frontier models now carry trillions of parameters, and their weights sit in HBM alongside the KV cache that holds each active request's context. Agentic workloads make it worse: an agent's trajectory accumulates context across repeated model calls and tool calls, and NVIDIA says some agent trajectories now process up to 3 million tokens. Retrieval-augmented generation adds enterprise documents on top. Meanwhile HBM supply is tight and prices are rising.

The blog's central number, which NVIDIA reports from its own testing and has not been independently verified: from Blackwell Ultra to the upcoming Vera Rubin platform, HBM capacity per GPU rises only about 3 percent, yet on the SemiAnalysis AgentX benchmark running a DeepSeek V4 Pro workload, throughput per dollar improves by up to 30x at 125 tokens per second per user when comparing Vera Rubin NVL72 to GB300 NVL72. NVIDIA makes the same style of claim one generation back — GB300 NVL72 delivers up to 10x the per-dollar throughput of the H200 platform at 100 tokens per second per user. The gap between 3 percent more memory and 30x more useful work is the entire thesis.

NVIDIA's analogy is a warehouse: capacity is the floor space, bandwidth is the loading docks. Once the docks saturate, extra floor space does nothing but cost money and draw power. Past that point — the memory wall — every concurrent user shares the same bandwidth, and added HBM goes to waste.

The engineering response attacks three pressure points. Model weights are compressed with the NVFP4 format, a 4-bit scheme that NVIDIA says recent models from Black Forest Labs, Thinking Machines Lab, Reflection, Meta and StepFun have already shipped checkpoints for, and within an NVL72 system weights are distributed across up to 72 GPUs so no single card carries the whole model. Applying NVFP4 to the KV cache as well shrinks its footprint compared with FP8, keeping more context resident.

The Vera CPU handles the tool-call problem — when an agent pauses to run a tool, its KV cache would otherwise sit idle in scarce HBM. Vera's faster tool execution shortens those idle windows, and its large LPDDR memory, connected to Rubin GPUs over NVLink-C2C, gives paused context a nearby place to wait. For longer absences, NVIDIA's Dynamo inference software moves context to CMX context memory storage, a new shared tier built for capacity at a far lower cost per bit than HBM. Unlike local SSDs it spans servers, and unlike enterprise storage it skips long-term data protection that short-lived context does not need. Partners listed for CMX include Cloudian, DDN, Dell Technologies, Pure Storage, HPE, IBM, NetApp, VAST and WEKA.

RAG gets its own treatment: NVIDIA's Nemotron Retriever models produce smaller embeddings with higher retrieval accuracy, and the cuVS library lets the vector index live outside HBM entirely, so only the passages a model actually needs ever enter scarce GPU memory. Dynamo's KV Cache Reuse ties the tiers together, managing cache as one system-wide resource so context computed once can be reused across GPU memory, CPU memory and storage.

The blog landed on a bruising day for the company — NVIDIA shares fell more than 9 percent intraday Thursday amid a broad selloff that knocked roughly $1.4 trillion off the stock's October peak, as investors reassess AI infrastructure payback periods. The timing made the message pointed: the industry's next phase is not about who buys the most memory, but who extracts the most useful tokens from every watt and every byte. All benchmark figures in the post are NVIDIA's own; independent numbers will have to wait for Vera Rubin hardware to reach customers.