Introducing go-cdc-chunkers: chunk and deduplicate everything · Plakar | The Open Standard for Backup and Restore
Search...
Book a demo<br>Demo
Book a demo<br>Demo
Introducing go-cdc-chunkers: chunk and deduplicate everything
Gilles Chehade
3978 words<br>July 11, 2025
19 min read
Edit this page on GitHub
You liked this article?
Share it
Join us
Help us
Gilles Chehade<br>CTO @ Plakar Korp, R&D software engineer
TL;DR:
Modern data systems suffer from redundancy—wasting time, compute, bandwidth and<br>storage on duplicate content. Traditional methods of compression don’t help<br>enough, especially when you need cross-file, shift-resilient deduplication in<br>large, encrypted datasets.
That’s why we built and released<br>go-cdc-chunkers: an open source<br>and ISC-licensed high-performance Go package for Content-Defined Chunking (CDC),<br>optimized for deduplication and resilience against data shifts.
Unlike traditional compression, CDC enables fine-grained, shift-resilient<br>deduplication, ideal for a wide range of uses including backup, synchronization,<br>storage, and distributed systems.
The problem of duplication
Every time your system moves, stores, or processes duplicated data, it’s doing<br>work it doesn’t need to. That means longer sync times, higher cloud egress fees,<br>bloated containers, over-provisioned caches, and users waiting for things that<br>should’ve been instant. Multiply that by thousands of files, logs, messages, or<br>binary blobs—and the inefficiency compounds rapidly. The more data you touch,<br>the more painful and expensive that duplication becomes.
In our business, where we need to process large amount of data, transfer it and<br>store it for extended periods of time, duplication is a nightmare: it increases<br>processing time, compute resources usage, transfer time and cost, pressure on<br>storage and space needed. Duplication wastes time and money at every - single -<br>step.
The solution ? Deduplication.
Deduplication isn’t just for backups. It’s for anything that handles recurring<br>or repetitive data: real-time collaboration tools, object storage systems, build<br>artifact pipelines, CI/CD caches, logging infrastructures, messaging queues,<br>document editors, and package registries. If your users upload revisions, move<br>large files across services, or repeatedly generate similar outputs, you’re<br>likely storing and reprocessing the same data again and again—sometimes<br>byte-for-byte.
By deduplicating at the right layer—whether file-level, block-level, or<br>chunk-level—you avoid wasting resources on what’s already known. You free up CPU<br>cycles for meaningful computation, reduce latency across your stack, shrink your<br>operational footprint, and make your systems leaner and faster. And if you’re<br>paying per gigabyte, per operation, or per millisecond? You’re literally buying<br>back time and money.
Here comes the go-cdc-chunkers package
To help developers build smarter, leaner systems that avoid redundant work,<br>we’re releasing<br>v1.0.0 of<br>go-cdc-chunkers—an open-source,<br>ISC-licensed library for high-performance Content-Defined Chunking (CDC) in Go.
It provides a framework to easily support new algorithms as research advances in<br>the field, and provides implementations for several algorithms including our<br>optimized version of FastCDC, our Keyed variant of FastCDC (discussed in this<br>post), an implementation of the JumpCondition optimization and even the more<br>recent UltraCDC.
This package is designed to make it easy to slice data into variable-sized,<br>content-aware chunks that are resilient to shifts and edits—perfect for<br>deduplication, delta encoding, change tracking, and more.
Whether you’re building synchronization tools, blob stores, data pipelines, or<br>just want to avoid wasting time and compute on repeated data, go-cdc-chunkers<br>gives you the primitives you need to chunk content efficiently and predictably.
Algorithm<br>Nanoseconds per operation<br>Throughput
Restic_Rabin<br>1932542209 ns/op<br>555.61 MB/s
Askeladdk_FastCDC<br>579593250 ns/op<br>1852.58 MB/s
Jotfs_FastCDC<br>448508056 ns/op<br>2394.03 MB/s
Tigerwill90_FastCDC<br>377360430 ns/op<br>2845.40 MB/s
Mhofmann_FastCDC<br>572578979 ns/op<br>1875.27 MB/s
PlakarKorp_FastCDC<br>117534472 ns/op<br>9135.55 MB/s
PlakarKorp_KFastCDC<br>115304560 ns/op<br>9312.22 MB/s
PlakarKorp_UltraCDC<br>79441967 ns/op<br>13516.05 MB/s
PlakarKorp_JC<br>49784102 ns/op<br>21567.97 MB/s
It’s very fast, very memory-conscious, and production-ready, with a<br>clean API that fits into streaming and batch workflows alike. We’re releasing it<br>not just as part of our internal stack, but as a practical tool for any<br>developer who needs data to be handled smartly—only once, not over and over.
oh… and it’s trivial to use:
chunker, err := chunkers.NewChunker("fastcdc", rd)<br>if err != nil {<br>log.Fatal(err)
offset := 0<br>for {<br>chunk, err := chunker.Next()<br>if err != nil && err != io.EOF {<br>log.Fatal(err)
chunkLen := len(chunk)<br>fmt.Println(offset, chunkLen)
if err == io.EOF {<br>// no more chunks to read<br>break<br>offset +=...