On March 26, SK Hynix fell 6.2 percent. Samsung dropped 4.7 percent. Micron lost 10 percent of its value in a single session. Across the memory chip sector, more than $90 billion in market capitalisation was erased in two days. The catalyst was a Google Research blog post about a new LLM inference algorithm called TurboQuant.

The selloff was a misread. But the underlying question about what AI efficiency improvements actually mean for hardware procurement deserves a careful answer.

TurboQuant, published at ICLR 2026 (arXiv:2504.19874), compresses the KV cache during LLM inference. Inference is the phase when a model is actively running and generating responses, as opposed to training, when it learns from data. The KV cache is a block of GPU memory that stores a running record of everything a model has processed in a given session; in long-context applications, it can grow to exceed the size of the model’s trained parameters. TurboQuant reduces it by up to 6x with no measurable accuracy loss, requiring no retraining and no changes to the model itself.

That is a real gain. What it is not is a general-purpose memory efficiency breakthrough. A model’s trained parameters, or weights, are unaffected. Training runs, which require massive amounts of high-bandwidth memory (HBM, the specialised fast-memory chips made by SK Hynix, Samsung, and Micron that sit inside AI accelerators), are entirely out of scope. The market reacted as if Google had found a way to make AI less hungry for memory across the board. The paper makes no such claim.

For data centres, the practical implication, once TurboQuant ships in production inference software, is more specific: better GPU memory utilisation on long-context workloads. Inference clusters built for document Q&A systems, retrieval-augmented generation pipelines (where a model queries a document library before responding), and multi-turn agentic workloads could serve more concurrent sessions on the same hardware. That changes the arithmetic on inference cluster sizing. It does not change training infrastructure procurement, and it does not reduce the memory floor set by model weights.

There is also a secondary effect that has received less attention. Aggressive KV cache compression tends to shift pressure from GPU memory to the storage tier. As compressed cache is offloaded to fast NVMe solid-state storage rather than held in GPU memory, the storage I/O architecture of inference nodes becomes more consequential. That is a different procurement story from the one the chip selloff implied.

On the chipmaking question, the strongest counterargument to the bear case is historical. Every major inference efficiency improvement has expanded AI deployment faster than it has reduced per-deployment hardware consumption, an effect economists call the Jevons Paradox: when a resource becomes cheaper to use, total consumption tends to rise rather than fall. TrendForce’s Q1 2026 report projects HBM contract prices rising 55 to 60 percent quarter-on-quarter, with demand tied specifically to long-context and multi-agent workloads.

TurboQuant is not yet in production. Official integration with vLLM and SGLang, the open-source frameworks that power most large-scale LLM inference deployments, is in progress but not complete. Benchmarks cover models up to 8 billion parameters; the zero-accuracy-loss claims have not been validated at the 70B-plus scale most enterprises deploy. The algorithm is real and the efficiency gains are well-founded. The $90 billion selloff was not.

Sources: Liu et al., arXiv:2504.19874; TrendForce Q1 2026 Memory Report

–
By the Control Plane Editorial Team