The CPU Cost of Protobuf Varints in Go
Newer|Older
2026-08-04<br>• 1748 words<br>• 9 minutes<br>• #protobuf<br>• #go<br>• #performance<br>• #software-architectureThe CPU Cost of Protobuf Varints in Go
Do fixed-size integers serialize faster than varints? We benchmark the CPU overhead of continuation-bit parsing using Go, vtprotobuf, and hyperpb.
When you define an integer field in a Protocol Buffers schema, int64 is a common default. Varint encoding compresses small numbers into a byte or two, keeping network payloads lean.<br>However, that compression comes at the cost of CPU cycles. To read or write a varint, the CPU must process the value byte by byte, checking continuation bits and shifting payloads. When a field sits in a high-throughput backend service, CPU efficiency often matters far more than saving a few wire bytes.<br>Protobuf also provides fixed-size integers (fixed32, fixed64, sfixed32, sfixed64), which use a constant-width, little-endian format. While taking more bytes for small values, their CPU path is dramatically simpler.<br>In Go benchmarks across standard google.golang.org/protobuf, PlanetScale vtprotobuf, and hyperpb, fixed-size integers prove up to 4.5x faster to encode and decode for packed 64-bit arrays—especially when values are large or negative. Here is how wire formats, CPU overhead, and runtime implementations interact in practice.<br>How the Wire Formats Actually Differ<br>Protobuf integer types divide into three encoding groups:<br>Standard varints: int32, int64, uint32, and uint64<br>ZigZag varints: sint32 and sint64<br>Fixed-size integers: fixed32, fixed64, sfixed32, and sfixed64<br>Standard Varints (int32 / int64)<br>Varints use protobuf’s Base 128 Varint format. Each byte reserves its MSB as a continuation flag, leaving 7 bits for payload:<br>Small numbers () fit in 1 byte.<br>Larger numbers require up to 10 bytes for 64-bit integers.<br>for v >= 17 {<br>buf[idx] = byte(v&0x7f | 0x80)<br>v >>= 7<br>idx++<br>buf[idx] = byte(v)
The decoder reverses this bit by bit. While negligible for scalars, this loop adds noticeable overhead over millions of elements in hot paths.<br>Negative numbers are particularly penalizing: two’s-complement representation sets bit 63. Because Base 128 packs only 7 bits per byte, negative integers encoded as standard int32/int64 force the maximum 10-byte encoding every single time—maximizing both wire size and CPU decoding cycles simultaneously.<br>ZigZag Varints (sint32 / sint64)<br>ZigZag encoding solves this penalty by mapping signed integers to unsigned values (0 -> 0, -1 -> 1, 1 -> 2, -2 -> 3), keeping small absolute values small on the wire.<br>However, ZigZag only solves payload bloat, not CPU cost. The parser still runs the varint continuation loop for every byte.<br>Fixed-Size Integers (fixed / sfixed)<br>Fixed-size integers skip small-value compression entirely:<br>fixed32 / sfixed32: 4 bytes<br>fixed64 / sfixed64: 8 bytes<br>Represented as raw little-endian values, the parser reads them directly without continuation checks or bit assembly. In Go schemas, fixed32/fixed64 map to uint32/uint64, while sfixed32/sfixed64 map to int32/int64.<br>The Benchmark Setup<br>To measure the practical difference in Go, I set up a test module with schemas containing packed repeated integer fields. Each test message holds 1,000 elements.<br>I benchmarked three value distributions:<br>Small Positive : integers in the range [0, 99]<br>Large Positive : integers in the range [2^50, 2^50 + 999]<br>Negative : integers in the range [-100, -1]<br>The tests evaluate three Go parsing implementations:<br>The standard google.golang.org/protobuf runtime using proto.Marshal and proto.Unmarshal<br>Generated marshal and unmarshal code from PlanetScale’s vtprotobuf plugin<br>Descriptor-based dynamic parsing using hyperpb with a reusable hyperpb.Shared memory arena<br>Note that hyperpb is not a direct drop-in replacement for standard struct unmarshaling. It evaluates how a specialized dynamic parser with zero-allocation memory arenas handles the wire formats, highlighting how parser architecture interacts with payload size.<br>All tests ran on an Apple M1 Pro (darwin/arm64) using Go 1.26.3. Averages represent 5 independent runs of 5 seconds each:<br>go test -bench=. -benchmem -benchtime=5s -count=5 > results.txt
Because these benchmark messages use packed repeated fields, each serialized payload consists of a single field tag, a length prefix, and the concatenated binary values. This structure amortizes the tag overhead across all 1,000 elements, isolating the actual cost of the integer serialization.<br>Wire Size Comparison<br>Before examining CPU timing, look at the serialized payload sizes for 1,000 integers:<br>Integer TypeSmall PositiveLarge PositiveNegativeint64 (Varint) 1,003 B 8,003 B10,003 Bsint64 (ZigZag Varint) 1,363 B8,003 B1,363 B sfixed64 (Fixed-Size) 8,003 B8,003 B 8,003 BZigZag (sint64) is slightly larger than plain int64 for small positive numbers because the bitwise mapping shifts positive values upward. Numbers above 63 cross into 2-byte varint...