PEP 784 – Adding Zstandard to the standard library | peps.python.org
Following system colour scheme
Selected dark colour scheme
Selected light colour scheme
PEP 784 – Adding Zstandard to the standard library
PEP 784 – Adding Zstandard to the standard library
Author:<br>Emma Harper Smith<br>Sponsor:<br>Gregory P. Smith<br>Discussions-To:<br>Discourse thread<br>Status:<br>Final<br>Type:<br>Standards Track<br>Created:<br>06-Apr-2025<br>Python-Version:<br>3.14<br>Post-History:<br>07-Apr-2025<br>Resolution:<br>25-Apr-2025
Table of Contents<br>Abstract
Motivation
Rationale<br>Introduction of a compression package
Implementation based on pyzstd
Minimum supported Zstandard version
Specification<br>The compression namespace
The compression.zstd module
libzstd optional dependency
Other compression modules
Backwards Compatibility
Security Implications
How to Teach This
Reference Implementation
Rejected Ideas<br>Name the module zstdlib and do not make a new compression namespace
Introduce an experimental _zstd package in Python 3.14
Introduce a standard library namespace instead of compression
Include zipfile and tarfile in compression
Do not include gzip under compression
Copyright
Important
This PEP is a historical document. The up-to-date, canonical documentation can now be found at compression.zstd.
See PEP 1 for how to propose changes.
Abstract
Zstandard is a widely adopted, mature, and highly efficient compression<br>standard. This PEP proposes adding a new module to the Python standard library<br>containing a Python wrapper around Meta’s zstd library, the default<br>implementation. Additionally, to avoid name collisions with packages on PyPI<br>and to present a unified interface to Python users, compression modules in the<br>standard library will be moved under a compression.* package.
Motivation
CPython has modules for several different compression formats, such as<br>zlib (DEFLATE), gzip, bzip2, and<br>lzma, each widely used. Including popular compression algorithms<br>matches Python’s “batteries included” philosophy of incorporating widely useful<br>standards and utilities. lzma is the most recent such module, added in<br>Python 3.3.
Since then, Zstandard has become the modern de facto preferred compression<br>library for both high performance compression and decompression attaining high<br>compression ratios at reasonable CPU and memory cost. Zstandard achieves a much<br>higher compression ratio than bzip2 or zlib (DEFLATE) while decompressing<br>significantly faster than LZMA.
Zstandard has seen widespread adoption in many different areas of computing.<br>The numerous hardware implementations demonstrate long-term commitment to<br>Zstandard and an expectation that Zstandard will stay the de facto choice for<br>compression for years to come. This is further evidenced by Zstandard’s IETF<br>standardization in RFC 8478 . Zstandard compression is also implemented in<br>both the ZFS and Btrfs filesystems.
Zstandard’s highly efficient compression has supplanted other modern<br>compression formats, such as brotli, lzo, and ucl due to its highly<br>efficient compression. While LZ4 is still used in very high throughput<br>scenarios, Zstandard can also be used in some of these contexts.<br>While inclusion of LZ4 is out of scope, it would be a compelling future<br>addition to the compression namespace introduced by this PEP.
There are several bindings to Zstandard for Python available on PyPI, each with<br>different APIs and choices of how to bind the zstd library. One goal with<br>introducing an official module in the standard library is to reduce confusion<br>for Python users who want simple compression/decompression APIs for Zstandard.<br>The existing packages can continue providing extended APIs or integrate<br>features from newer Zstandard versions.
Another reason to add Zstandard support to the standard library is to resolve<br>a long standing open issue (python/cpython#81276) requesting Zstandard<br>support in the tarfile module. This issue has the 5th most “thumbs up”<br>of open issues on the CPython tracker, and has garnered a significant amount of<br>discussion and interest. Additionally, the ZIP format standardizes a<br>Zstandard compression format ID, and integration with the zipfile<br>module would allow opening ZIP archives using Zstandard compression. The<br>reference implementation for this PEP contains integration with the<br>zipfile, tarfile, and shutil modules.
Zstandard compression could also be used to make Python wheel packages smaller<br>and significantly faster to install. Anaconda found a sizeable speedup when<br>adopting Zstandard for the conda package format:
Conda’s download sizes are reduced ~30-40%, and extraction is dramatically faster.<br>[…]<br>We see approximately a 2.5x overall speedup, almost all thanks to the dramatically faster extraction speed of the zstd compression used in the new file format.—Anaconda blog on Zstandard adoption
Zstandard has a significantly higher compression ratio compared to wheel’s<br>existing zlib-based compression, according to lzbench, a comprehensive<br>benchmark of many different compression...