Single-User Inference Card - Marko Haberl on AI

Marko Haberl on AI

SubscribeSign in

Single-User Inference Card<br>SUIC

Marko Haberl<br>Aug 16, 2026

Share

This is a write-up about a piece of hardware that does not exist yet. But it will, because we need it badly.<br>I will call it a Single-User Inference Card , or SUIC for short. Everyone likes a good acronym.<br>The idea for its usage is:<br>for AI inference (not for AI training and not for playing video games)

for a single user only (not serving 1,000 enterprise chatbot users from a datacenter)

The restrictions are:<br>no fantasy super-chips, we can use ONLY combinations of technology available today

it should not cost you much more than the PC you use right now

This is the thing you would plug into your PC, and you could have smarts comparable to SOTA models, but locally. Full privacy, full availability, at the cost of the electricity used for computation.

Building the card

For the compute base, I will pick 3 popular consumer NVIDIA cards, all released in 2025. Their launch prices ranged from $550 to $2,000.<br>For memory, I will pick something commercially available since 2024 called HBM3E.<br>It is mass-produced by SK Hynix, Micron, and Samsung. A six-pack of 36 GB stacks costs about $2,500.<br>What we do now is attach them to the cards.<br>In truth, that would require a modified GPU die with a new memory interface and different packaging. BUT none of the technology is new.<br>The GPU and HBM stacks need an interposer/package comparable to CoWoS. That currently costs about $750. Add another $500 for the special PCB, voltage regulators, and extra cooling.

Now that we’ve got our 3 SUIC cards, I want to benchmark them. But against what?<br>Choosing the model to test on

We could pick any of 100 different usage scenarios, but my gut feeling tells me software devs would be among the first customers. So agentic coding it is.<br>What I wanted to show was calculations on a dense model. That would be beautifully simple and would look super strong, but I promised you frontier smarts.<br>Claude Opus-category models like Kimi K3, GLM-5.2, DeepSeek-V4-Flash-0731, MiniMax M2.7 are all MoEs.<br>Best I found was Mistral Medium 3.5 128B, but that is just not in the same league. It’s more like Sonnet 4.5/4.6. In quality, it’s not much better than the 28.6 GB Qwen3.6-27B-Q8_0 that you can already run on RTX 5090.<br>I will proceed with the DeepSeek-V4-Flash-0731 .<br>My reasoning is that it has comparable benchmarks to the recently popular Opus 4.8 (no comparisons to Opus 5 published yet), and I found testing results that give me the exact measurements I need on a real MI300X card (with HBM3 memory) to estimate the performance of our SUIC 5070, SUIC 5080, and SUIC 5090 cards.<br>The reported model weights are 156.67 GiB (about 168 GB), which leaves us about 48 GB for KV cache and the other memory spenders.<br>MI300X numbers:<br>67.3 t/s native decode

152.6 t/s with DSpark-7 speculative decoding (a later run)

11,690 t/s uncached single-request prefill

These are the benchmarks I mentioned:

The ugly number here is NL2Repo. This measures performance on long-running tasks (”long-horizon”). You tell the agent what to build and check it in the morning. Opus 4.8 would do better. I guess Opus 5 or Fable would do even better than that. But... there will be a model in 2027 that beats those, and another one a year later. These numbers are a signal, not a true measure of what your experience will be.<br>For now, let’s just assume that you could do very good work with it, if you had it running locally as much as you wanted and if the performance was good enough .<br>Performance gains

OK, now on to the best part: fantasy numbers for fantasy hardware.<br>About the memory, and why does this all even work

Consumer cards RTX 5070, RTX 5080, RTX 5090 use GDDR7 memory.<br>Datacenter cards like MI300X, H100, H200, B300 use HBM3 or HBM3E memory.<br>Fun fact: GDDR7 is much faster per pin<br>We are talking like 30 vs 9 Gb/s. More than 3x!<br>The elephant in the room with today’s inference hardware is the bus width .

Math is super simple<br>Bandwidth = (pin speed x bus width in bits) / 8

HBM3E stacks are manufactured with a lot of pins. 1024 per each .<br>So our 6-pack has 6144-bit bus width.<br>Let’s do some multiplications (source):<br>9.375 Gb/s x 6144 bit / 8 = 7200 GB/s

All three SUICs get the same 216 GB memory capacity.<br>As for memory bandwidth. What we know from NVIDIA H100, H200, AMD’s MI355X and others is that the die can pull about 30-36 GB/s per SM. NVIDIA reports that B300 can pull 50 GB/s (8TB/s divided by 160 SM).<br>My guess is that is safe to assume that with our new design we can get 42.5 GB/s per SM so our 5090 can exactly pull the complete 7200 GB/s.

In general, especially with dense models, at low concurrency, decode will be memory-bound , while prefill is compute heavy. On the other hand MoE model that I chose is launch-bound as the author states or I would rather name that latency-bound .<br>Before we do the calculation, I just want to say a word...