A new memory technology called High Bandwidth Flash is being developed to address the growing memory demands of large language models during inference, using stacked NAND flash to increase read speeds and capacity while reducing costs
As large language models (LLMs) continue to expand in size and deployment, the memory required to serve them at scale has become a critical technical and economic constraint. Traditional solutions such as high bandwidth memory (HBM) and dynamic random-access memory (DRAM) offer the speed needed for both training and inference, but their cost and limited capacity have prompted memory manufacturers to explore alternatives. One such approach, High Bandwidth Flash (HBF), adapts the stacking and packaging techniques used in HBM to the more affordable and capacious NAND flash memory found in consumer devices like SD cards and thumb drives.
HBF is designed to address a specific challenge: the memory bottleneck encountered during AI inference, when a trained model's weights must be rapidly accessed to generate outputs for users. While NAND flash is known for slow write speeds, its read performance can be significantly improved through advanced packaging and vertical stacking. According to technical documentation from Sandisk, the first generation of HBF is expected to stack up to 16 NAND flash dies, reaching a total capacity of 512 GB per stack and delivering read bandwidth up to 1.6 TB/s. Future generations are projected to increase read bandwidth to 2 TB/s and 3.2 TB/s, respectively. These figures remain below the maximum bandwidth of HBM4E, which can reach 3.6 TB/s per stack, but HBF offers a substantial improvement over conventional flash storage and at a lower cost per byte.
Inference Versus Training
The suitability of HBF for AI workloads depends on the distinction between training and inference. Training large models requires frequent updates to billions or trillions of parameters, making high write bandwidth essential-a task for which flash memory is poorly suited. Inference, by contrast, involves reading fixed model weights to generate predictions, with little or no need for writes. This read-heavy pattern aligns with the strengths of HBF, allowing it to serve as a high-capacity, lower-cost memory tier for storing static model weights and precomputed data such as key-value caches. In this configuration, HBM or DRAM can be reserved for high-speed scratchpad operations, while HBF supplies the bulk of the model data.
Despite its promise, HBF is not yet commercially available. Sandisk and SK Hynix, two major memory manufacturers, have initiated a joint effort to standardize HBF through the Open Compute Project, a consortium that develops open hardware standards for data centers. The first kickoff event for this standardization process took place on 25 February 2026. No timeline has been set for publication of the standard, and broad deployment is likely several years away. Technical fact sheets and roadmaps have been released, but independent performance verification and large-scale operational testing have not yet been reported.
Technical and Economic Trade-Offs
While HBF cannot match the peak bandwidth of HBM, its potential to alleviate memory capacity bottlenecks is significant. By enabling data centers to store more model weights per accelerator, HBF could reduce the number of expensive accelerators required for large-scale inference, improving energy efficiency and lowering operational costs. This approach is positioned as complementary to HBM rather than a direct replacement, with HBF handling static, read-only data and HBM reserved for latency-critical operations. The non-volatile nature of NAND flash also means that data persists without power, further reducing energy consumption compared to DRAM-based solutions.
However, the adoption of HBF will depend on several unresolved factors. These include the reliability of stacked NAND flash under sustained high read loads, the integration of HBF into existing accelerator architectures, and the development of software and hardware standards to support seamless tiered memory management. The lack of independent benchmarking and the absence of real-world deployment data mean that claims about performance, cost savings, and energy efficiency remain provisional. Regulatory and procurement decisions in the data center industry may also influence the pace and scope of HBF adoption.
Numerical Context
According to Sandisk's published specifications, the first generation of HBF is expected to deliver up to 1.6 TB/s of read bandwidth per stack, with a maximum capacity of 512 GB per stack using 16 vertically stacked NAND flash dies. In comparison, HBM4E can reach up to 3.6 TB/s per stack, while DDR5 memory modules offer up to 70.4 GB/s per DIMM. These figures highlight the trade-off between bandwidth and capacity, with HBF positioned as a high-capacity, moderate-bandwidth solution for inference workloads. The technology remains in the pre-deployment phase, with no independent operational benchmarks available as of early 2026.
Understanding the distinction between training and inference is essential for evaluating memory technologies in AI. Training involves updating model parameters through repeated exposure to data, requiring high write and read bandwidth. Inference, by contrast, uses fixed model weights to generate outputs, making it primarily a read-intensive process. Memory solutions like HBF are tailored to this inference phase, where rapid access to large, static datasets is more important than write speed. The effectiveness of such tiered memory architectures will depend on their integration with accelerators, software frameworks, and data center infrastructure, as well as on independent validation of their claimed performance and reliability.
