NVIDIA Proposes Full-Stack AI Memory Co-Design, Delivering Up to 30x Inference Throughput Improvement

Odaily Report: NVIDIA announced that as AI models scale up, context windows lengthen, and agent applications become more widespread, GPU memory pressure continues to rise, HBM supply is tightening, and prices are increasing. NVIDIA proposes improving memory usage efficiency through co-design across GPU, CPU, networking, storage, and software, rather than simply increasing GPU memory capacity.

Specifically, NVIDIA compresses model weights and KV Cache using the NVFP4 low-precision format, offloads temporarily inactive context to lower-cost storage tiers, and frees up HBM space through multi-GPU distributed deployment.

NVIDIA disclosed that in SemiAnalysis AgentX testing, GB300 NVL72 achieved up to 10x the cost-per-inference throughput of the H200 platform under specific workloads; Vera Rubin NVL72 delivers up to 30x higher cost-per-inference throughput compared to GB300 NVL72, while single-GPU HBM capacity increases by only about 3%.