Release v1.2.0 · minio/minlz · GitHub
//releases/show" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
//releases/show;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
Uh oh!
There was an error while loading. Please reload this page.
minio
minlz
Public
Notifications<br>You must be signed in to change notification settings
Fork<br>17
Star<br>139
v1.2.0
Latest
Latest
Compare
Choose a tag to compare
Sorry, something went wrong.
Filter
Loading
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
No results found
View all tags
klauspost
released this
13 Jul 09:39
v1.2.0
6613957
This commit was created on GitHub.com and signed with GitHub’s verified signature .
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
feat: Search compressed data without decompression
Compression saves space, but it usually makes your data opaque: to find anything you first
have to decompress the whole thing. MinLZ changes that. A MinLZ stream can carry a small
per-block search index that lets mz search find a byte sequence while skipping every block
that provably can't contain it — those blocks are never decompressed, and (with a sidecar) never
even read from disk.
The result: you keep your data compressed and searchable.
One number to set the scene: finding a specific string in a 10 GB CockroachDB log
(compressed to 578 MB) takes 0.14 s with mz search, reading about 133 MB. Decompressing
that log and piping it to grep — the usual way to search compressed data — takes ~5 s and
reads the whole file. zstd -dc | grep takes ~5 s, lz4 -dc | grep ~10 s. Scanning the raw
10 GB with rg takes ~8 s.
The search index is stored as skippable chunks: older MinLZ readers ignore them and the compressed
payload is byte-for-byte unchanged, so adding search is always backward-compatible.
Search indexes can be built while compressing or after the fact as sidecar streams,
allowing them to be used separately from the compressed data.
How it works (in 30 seconds)
Every block gets a tiny bloom-filter table: the byte-windows in the block are hashed and their
bits set. To search, mz search hashes the same-length windows of your pattern and checks each
block's table:
any window bit missing → the pattern is definitely not in that block → skip it (no decode);
all bits present → the block might match → decode and scan it.
Longer/rarer patterns produce more independent window checks, so blocks that don't contain them are
rejected with near-certainty. Tables can live inline in the .mz, or in a sidecar .mzs
file you build afterwards — handy for data you can't or don't want to re-compress.
This is why it matches with compression. Compressible data is very likely to produce
search indexes that are also useful, meaning there is a reasonable reduction in the
search table compared to the raw data.
For full details: SEARCH.md (usage & tuning) and SPEC_SEARCH.md (wire format).
When to use it — and when not
We will use the command-line tool mz search to demonstrate, but everything is available via the Go API.
This is a specialized tool. It is dramatic when it fits and pointless when it doesn't, so read this
part first.
Reach for mz search when:
You search for literal byte strings — IDs, error codes, request paths, hostnames, JSON keys,
hashes. (Not regex. See below.)
The pattern is reasonably selective — it appears in a minority of blocks. Rare → huge win.
The data is large, archived, or remote and you'd rather not materialize the whole thing.
You search the same corpus repeatedly — build the index once, query many times.
You are I/O-bound: on object storage or a cold cache, skipped blocks issue zero reads.
You need to confirm a string is absent ("is this error anywhere in 10 GB?") — the fastest case
of all.
Don't bother when:
You need regex, case-insensitive, or fuzzy matching. mz search is literal bytes only.
The pattern is very common or shorter than the index's match length — little or nothing
gets skipped, and you pay a small table-check tax on top of a full decode.
It's a one-shot scan of small data that's already in RAM and you have ripgrep — rg is
superb there and hard to beat.
Rule...