Performance Has Layers | Oxide Computer Company
18 Jun 2026
Performance Has Layers
Steve Karam<br>Product Assurance
One of the most beneficial things about building the whole stack is also one of the toughest things: you also own all the problems. When a packet leaves a customer’s guest, travels through a virtual NIC, crosses our software switch, hits the OPTE data path, rides the physical fabric, and arrives at another guest, it passes through several distinct components.
And if you build all of them, as we do, then every one of those components is a place you can make the system faster. It beats the hell out of a five-way call with different vendor support teams.
But just because you can speed up those components, it doesn’t mean any single layer is “the” performance story. Each one helps a different kind of workload, manifests as a specific type of bottleneck, and, when cleared, reveals another layer of wait.
So let’s talk about a handful of these layers, what each one does, and why owning all of them is the thing that lets us keep building high performance on-prem infrastructure.
A specific need<br>Over our last couple of releases, we’ve focused heavily on the network; namely, the introduction of end-to-end IPv6 plumbing and jumbo frames. Introducing these to a self-contained rack involved a lot of decisions, along with thorough investigation of the expected outcomes. We started by looking at the layers and knobs we could influence. From the wire up:
The physical fabricLinks between racks and the switches that carry them.<br>Frame sizeHow many bytes ride in a single packet (the minimum transmissible unit, or MTU).<br>OffloadsWhether segmentation and checksums happen in the guest, or are handed off to underlying infrastructure that can do them more efficiently.<br>CPU placementWhere the virtual NIC’s worker threads run relative to the guest’s own vCPUs.<br>Flow parallelismHow many independent connections the work is spread across.<br>ProtocolIPv6 is the rack’s native underlay; the only question is whether a guest speaks IPv6 or IPv4 on top of it.
Each of these knobs helps one or more types of workload. The rest of this post walks them, roughly in the order a packet would encounter them, and notes what each one is good for.
Frames, and the wire that carries them<br>Start with the most visible knob, and the one that took quite a bit of work to turn: the size of a packet.
Size matters because the network does a roughly fixed amount of work per packet, mostly independent of how many bytes the packet carries. Every packet has its headers parsed at several layers. Every packet runs the gauntlet of firewall rules, route lookups, address resolution, and connection-tracking state, much of it in software in the data plane. So if you want to move a megabyte and each packet holds 1500 bytes, you pay that per-packet tax around 700 times. Raise the packet to 9000 bytes and you pay it roughly 120 times. Same data, one-sixth the overhead.
Inside a rack, we have always taken advantage of this. Guests advertise TCP segmentation offload (TSO), so a guest hands us buffers up to 64 KB at a time over virtio. We carry them across a 9000-byte underlay, set the segment size to leave room for encapsulation, and let the sled’s physical NIC slice the big buffer back into wire-sized packets on the way out. This is tunneled TSO, and it is why traffic inside a VPC has effectively been riding jumbo frames for a long time, and why internal throughput has always been high.
The conservative part was at the rack’s edge. Traffic leaving the rack travels onto networks we do not control, and the safe assumption for an arbitrary external path is the Internet-standard 1500-byte MTU. So that is what we advertised to guests for external traffic.
The major public clouds make the same choice, but it is worth being precise about why, because the word “external” means something very different to them than it does to us. For a hyperscaler, the internal network is an entire availability zone: a building, or several, stitched together by a fabric they own. Almost everything a workload talks to lives inside that boundary, and the only thing that is genuinely “external” is the public Internet. Falling back to a 1500-byte MTU at that edge costs them little, because the edge is far away and rarely on the fast path.
For a single Oxide rack, the boundary sits much closer in. “External” is everything past one rack, and a lot of what lives there is not the public Internet at all: it is the rack in the next row, reached over a fabric the same operator owns and provisioned. That operator often does control the upstream and knows exactly what it can carry. Applying the hyperscaler’s conservative default to that link treats a known, fat, operator-owned path as if it were the open Internet, and leaves a great deal of performance on the table.
External jumbo frames let those operators opt out of the conservative default. The end-user-visible part is a single boolean on an...