New CMX Hardware Aims to Smash AI's Growing Memory Bottleneck

MILPITAS, CA – March 13, 2026 – As artificial intelligence races towards more sophisticated, human-like interactions, the very hardware that powers it is hitting a wall. Now, a new partnership between server solution leader AIC and storage technology firm ScaleFlux, supercharged by NVIDIA's latest networking innovations, aims to tear that wall down with a purpose-built hardware platform for what they call Context Memory Storage (CMX).

The joint solution, announced this week, directly targets a critical and escalating bottleneck in AI infrastructure: the massive memory requirements of advanced AI models. By combining the AIC F2032-G6 storage system with ScaleFlux NVMe SSDs and NVIDIA's BlueField-4 DPU, the companies are introducing a new architectural tier designed to manage the exploding data generated during AI inference, particularly for long-context and agentic AI applications.

The AI Memory Crisis

At the heart of the problem is a component of large language models (LLMs) known as the Key-Value (KV) cache. When an AI model generates a response, it must remember the context of the entire conversation or document. To do this efficiently, it stores key data points from previous tokens in this KV cache, preventing the need for constant, costly recalculation.

However, this cache grows linearly with the length of the interaction. For multi-turn conversations, complex reasoning tasks, or autonomous AI agents that must maintain context over long periods, the KV cache can swell to dozens of gigabytes per user. This memory consumption can quickly overwhelm the expensive, high-bandwidth memory (HBM) on the GPUs themselves, which is the lifeblood of AI processing. When GPU memory is full, the system grinds to a halt, forcing operators to either limit the number of simultaneous users, shorten the context, or offload the data to slower system memory—all of which lead to underutilized, multi-million-dollar GPUs and a poor user experience.

This bottleneck directly impacts the "time to first token" (TTFT), a critical metric for how responsive an AI application feels. The longer it takes to process the initial prompt and its context, the more sluggish the AI seems. In the race to deploy powerful AI services, this memory crisis has become a central challenge for infrastructure operators.

A New Tier in the AI Data Center

Rather than relying on software workarounds alone, AIC and ScaleFlux are proposing a fundamental architectural shift. Their solution establishes CMX as a new, distinct hardware layer in the AI data center, sitting between the ultra-fast GPU memory and slower, traditional storage.

"AI inference is rapidly shifting from stateless queries to persistent, long-context interactions," said Michael Liang, CEO at AIC, in the announcement. "Our new F2032-G6 platform, combined with BlueField-4 and ConnectX-9 networking, provides the high-performance storage architecture needed to support context memory storage at scale."

The platform is built on AIC’s F2032-G6, a high-density "Just a Bunch Of Flash" (JBOF) system. This 2U chassis is populated with ScaleFlux’s specialized NVMe SSDs, which are engineered to handle the high-input/output, low-latency access patterns characteristic of KV cache workloads. The result is a shared pool of extremely fast storage that can hold and serve large context datasets to entire clusters of GPUs, effectively extending their memory capacity at a fraction of the cost of adding more HBM.

"Context memory is emerging as a new data tier in AI infrastructure," stated Hao Zhong, CEO and Co-Founder of ScaleFlux. "By pairing ScaleFlux NVMe SSDs with AIC's high-density JBOF platform and NVIDIA's advanced data-center networking technologies, we are delivering a hardware solution optimized for the next generation of AI inference pipelines."

The NVIDIA Connection

The linchpin of this new architecture is the integration of NVIDIA's latest data center technologies. The AIC system incorporates the NVIDIA BlueField-4 Data Processing Unit (DPU) and the ConnectX-9 SuperNIC. These are not mere networking cards; they are powerful co-processors designed to offload the immense burden of managing network, storage, and security traffic from the main server CPUs and, by extension, the GPUs.

The BlueField-4 DPU, which NVIDIA positions as the "operating system for AI factories," provides a staggering 800 Gb/s of throughput. Within the CMX platform, it manages the flow of KV cache data between the GPU servers and the flash storage, ensuring that the transfer is executed with minimal latency and without interrupting the main processors. This allows the GPUs to remain focused on their primary task: AI computation. This integration aligns perfectly with NVIDIA's broader vision for an Inference Context Memory Storage (ICMS) platform, turning the KV cache into a shared, high-bandwidth resource across an entire AI pod.

Maximizing AI's Return on Investment

Beyond the technical innovation, the CMX solution addresses a pressing business imperative: maximizing the return on investment (ROI) from colossal AI infrastructure expenditures. GPU clusters represent investments of tens or even hundreds of millions of dollars. Every microsecond a GPU sits idle waiting for data is money down the drain.

By providing a high-performance buffer for context memory, the AIC and ScaleFlux platform aims to keep GPUs fed with data, boosting their utilization rates. Higher utilization means more inference jobs can be run on the same hardware, accelerating time-to-market for new AI services and lowering the total cost of ownership.

While other solutions like software-based cache offloading and Retrieval-Augmented Generation (RAG) exist, the CMX approach advocates for a dedicated, hardware-accelerated tier. The argument is that for the most demanding, at-scale deployments, a purpose-built hardware solution offers a level of performance, predictability, and efficiency that software-only methods struggle to match.

As organizations deploy increasingly autonomous and conversational AI agents, the need for scalable and efficient context management will only intensify. This new hardware platform from AIC and ScaleFlux represents a significant step in building the foundational infrastructure required to support this next wave of artificial intelligence, ensuring that the systems of tomorrow are not constrained by the memory limitations of today.