GPUs could explode to multiple TB with new storage-inspired memory tech

Jump to main content

Search

REG AD

STORAGE

GPUs could explode to multiple TB with new storage-inspired memory tech

High-bandwidth flash promises SSD-like capacities with HBM-like speeds, but it's not all unicorns and rainbows

Tobias Mann

Tobias<br>Mann

SYSTEMS EDITOR

Published<br>thu 30 Jul 2026 // 19:33 UTC

Virtually every high-end GPU and AI accelerator relies on high bandwidth memory (HBM), which can shuffle data around at multiple terabytes a second but can only reach into the gigabytes, with models often needing to be shared across multiple processors. However, an emerging storage technology could change that, boosting accelerator memory capacity from hundreds of gigabytes to terabytes.<br>The technology, called high-bandwidth flash (HBF), is being developed by Sandisk and SK Hynix and aims to provide SSD-like capacities at HBM-like speeds.<br>Peeling back HBF’s layers

REG AD

Conceptually, high-bandwidth flash looks and sounds a lot like HBM. It’s assembled by stacking multiple layers (16 in the case of Sandisk’s first-gen modules) of memory together, which boosts capacity and bandwidth. But where HBM uses DRAM, HBF aims to use NAND flash.

REG AD

Sandisk claims its first generation of high-bandwidth flash will supposedly achieve read bandwidths up to 1.6 TB/s [PDF], making it a bit faster than HBM3e but significantly slower than HBM4, which is already hitting 2.5 TB/s per 12-high stack. Future HBF generations are expected to push bandwidth to over 2 TB/s and eventually 3.2 TB/s.<br>While bandwidth makes HBF interesting as an alternative to HBM, its real party trick is capacity. Because it’s built using NAND, Sandisk says it can achieve capacities up to 256 Gb per die, which translates to 512 GB per 16-high module. That’s more than 14 times the capacity of the HBM4 used in AMD and Nvidia’s latest accelerators.<br>Continuing with the similarities, HBF modules share similar packaging requirements to HBM, which means you can expect them to be fused to the GPU die using advanced packaging techniques like TSMC’s CoWoS, or Intel’s EMIB and Foveros tech. Nothing particularly exotic as AI accelerators go.<br>What’s more, the storage vendor doesn’t expect the modules to come at a power or price premium over HBM. And from a bits per dollar standpoint, HBF looks like a stellar option.<br>If all this sounds a bit too good to be true, that’s because for all of HBF’s benefits, it comes with some rather significant compromises.<br>NAND still isn’t DRAM<br>The main trade off, as we understand it, is write endurance and access latency. HBF may perform like HBM on paper, but it’s still using NAND, which has a finite write endurance before it wears out and has access latencies measured in microseconds as opposed to tens of nanoseconds for DRAM.<br>If you were to swap HBM for HBF, it (probably) wouldn’t perform very well and it’d wear out pretty quickly, rendering that $50,000-plus GPU of yours a paperweight — not ideal for a product that’s being asked to serve longer to suit hyperscalers' depreciation schedules.

REG AD

Instead, Sandisk and SK Hynix propose using HBF to supplement HBM to make inference more cost effective. HBM handles all the write intensive stuff while HBF takes care of the read heavy parts of the pipeline.<br>While we talk about inference as one job, it's really a collection of many that can broadly be broken into two categories, one that’s compute intensive and another that’s bandwidth bound.<br>The first of these phases, called prefill, involves tokenizing and embedding prompts, feeding them through the model in one big forward pass, generating the key value caches used to track state, and outputting the first token.<br>The second, called decode, reads the entirety of the model's weights, or in the case of a mixture of experts (MoE) models, their active parameters, from memory over and over again for each token generated. Because of this, how quickly an AI system can churn out tokens is directly proportional to how fast its memory is.<br>Prefill is comparatively write heavy, so it makes sense to do as much of that in HBM as possible. But the decode phase is almost entirely read, which makes HBF an ideal medium for storing model weights as write endurance really isn’t a factor.<br>It becomes a sort of write-once, read-many scenario, which is perfect for NAND flash since reads are essentially free. You could almost think about HBF a bit like a rewritable ROM cartridge for models. And because HBF is non-volatile, it becomes a bit like Intel’s Optane persistent memory. There is no need to wait for weights to reload from storage into GPU memory; they’re already there and ready to go.<br>Sandisk’s slides propose a couple of different options including one that would feature 3.12 TB of memory across two stacks of HBM and six stacks of HBF.

MORE CONTEXT

Samsung warns memory crunch will last through 2028 as profit rises 19-fold

Qualcomm won’t be a big datacenter player...