Etched Sohu vs NVIDIA: Transformer ASIC vs GPU (2026) | Spheron Blog

XDiscordLinkedInShare

Etched Sohu is a transformer-only ASIC, and Etched AI claims one 8-chip Sohu server delivers 500,000 tokens per second on Llama 70B, roughly 62,500 tokens/sec per chip. For comparison, a single H100 SXM5 achieves around 700 tokens/sec at batch 1 with vLLM. That per-chip advantage is real in the sense that it reflects Sohu's architecture: the chip hard-codes transformer attention directly into silicon as fixed-function logic rather than as software running on a programmable compute unit. The implied tradeoff is the entire story here. Sohu is a bet that transformer attention is the dominant AI architecture for the next several years, and that the workload is stable enough to justify giving up all programmability.<br>For teams evaluating inference hardware right now, the practical question is not whether Sohu is fast. It is whether the architectural constraints, supply risk, and toolchain migration cost are acceptable for your specific workload. This post covers the architecture in detail, compares Sohu against H100, B200, and the Groq 3 LPU, and gives a framework for deciding when the ASIC bet pays off. For a currently available ASIC comparison, see the SambaNova SN40L vs H200 and B200 guide, which covers the RDU architecture and live cost-per-token math.<br>Update: Etched Exited Stealth on June 30, 2026<br>Etched formally came out of stealth on June 30, 2026, and the announcement moved the story from "interesting claim" to "funded, contracted roadmap." The concrete facts: working A0 silicon demonstrated, a rack-scale 8-chip Sohu inference system unveiled, roughly $800M raised across four rounds (including a reported $500M round at a $5B valuation), over $1B in signed customer contracts, and first racks scheduled to ship in summer 2026. What has not changed: no independent third-party benchmarks, no public pricing, and no self-serve way to rent a Sohu today. Everything below, including the cost-per-token framework, still applies; the difference is that the availability question now has a date attached instead of a shrug.<br>What Is Etched AI and the Sohu Chip<br>Etched AI is a chip startup founded in 2022, having raised roughly $800 million across four rounds, including a reported $500M round at a $5B valuation. Their first product is the Sohu chip, a transformer-only ASIC designed for autoregressive language model inference. The name and funding are real. Since the June 30, 2026 stealth exit, Etched has shown working A0 silicon and booked over $1B in signed contracts, with first racks slated for summer 2026. It is still not something you can buy or rent today, and no independent benchmarks exist yet.<br>The core architectural claim is that by implementing transformer attention as fixed-function silicon rather than as programmable matrix multiply instructions, Sohu can achieve throughput figures that no GPU can match for the same workload class. NVIDIA GPUs are programmable compute units that run CUDA kernels written in software. Groq's LPU is a dataflow processor with a custom compiler. Sohu takes a different position: there is no programmability layer at all. The chip does one thing, and it does it by being physically built for that one thing.<br>This is different from other custom inference chips in an important way. The Groq 3 LPU has a compiler that can, in principle, be extended. AMD GPUs support ROCm. Tenstorrent ships TT-Metal as open-source software. Sohu's architecture does not have a software abstraction layer in the same sense. If transformer attention changes in the next generation of models, the chip cannot adapt. For hyperscaler-built ASICs (Trainium 3, Maia 200, Meta MTIA), see the hyperscaler custom AI chip roundup.<br>Sohu Architecture: The Transformer-Only Bet<br>Fixed-Function Transformer Units<br>Transformer attention requires three core operations at each layer: computing query, key, and value projections; running multi-head attention across the KV cache; and passing results through a feed-forward network. On a GPU, each of these is a CUDA kernel that can be swapped out for a different implementation. PagedAttention, FlashAttention-2, and FlashAttention-3 are all software optimizations that improve how attention is computed on general-purpose hardware.<br>On Sohu, these operations are hardwired as static circuits. The chip does not have a general matrix multiply unit that runs attention software. It has physical circuits that implement the attention computation directly. This removes all of the overhead from kernel launch latency, memory allocation, and scheduler decisions. It also means the chip cannot run any computation that does not map to transformer attention. There is no way to compile a convolution, an SSM scan, or a diffusion U-Net step onto Sohu because there are no programmable units to target.<br>On-Chip Memory Design<br>Sohu's throughput advantage comes primarily from memory bandwidth. The bottleneck for autoregressive...