zstd-rs

zstd

This library is a rust binding for the zstd compression library.

Documentation

1 - Add to cargo.toml

$ cargo add zstd

# Cargo.toml

[dependencies]
zstd = "0.14"

2 - Usage

This library provides Read and Write wrappers to handle (de)compression, along with convenience functions to made common tasks easier.

For instance, stream::copy_encode and stream::copy_decode are easy-to-use wrappers around std::io::copy. Check the stream example:

use std::io;

// This function use the convenient `copy_encode` method
fn compress(level: i32) {
    zstd::stream::copy_encode(io::stdin(), io::stdout(), level).unwrap();
}

// This function does the same thing, directly using an `Encoder`:
fn compress_manually(level: i32) {
    let mut encoder = zstd::stream::Encoder::new(io::stdout(), level).unwrap();
    io::copy(&mut io::stdin(), &mut encoder).unwrap();
    encoder.finish().unwrap();
}

fn decompress() {
    zstd::stream::copy_decode(io::stdin(), io::stdout()).unwrap();
}

Asynchronous support

The async-compression crate provides an async-ready integration of various compression algorithms, including zstd-rs.

Performance

Which function to use

The frame header can record how big the data will be once decompressed, and whoever reads it can then size their buffer in one go instead of growing it - worth roughly a factor of two to them. Only a compressor that knows the length up front can record it:

you are compressing	into a Vec	into a Write
a slice or a Vec	bulk::compress	compress_sized_into
a file	compress_file	compress_file_into
a Read whose length you know	compress_sized	compress_sized_into
any other Read	encode_all	copy_encode

Everything but the last row records the size, whatever it is. The last row records it when the source turns out to be shorter than 128 KiB, and leaves it out beyond that. Reach for the Write column when the data is large enough that you would rather not hold a Vec of it - these stream straight through, so only the compressor's own buffers are in memory. For a source that is neither a file nor a slice, its length is the one thing you have to supply yourself.

On the way back, decode_all reads that header and sizes its output for you, and decompressed_size gives you the same number for a buffer of your own.

Matching the C library

This wrapper adds little over the C library. On the same data at level 3, a re-used bulk::Compressor and bulk::Decompressor land within a few percent of what zstd -b3 reports for the same file. If you are seeing much less than that, it is usually one of these:

• A debug build. Always measure with --release; a debug build times this crate's wrappers rather than zstd.
• Growing a Vec rather than sizing it. The convenience functions that return a Vec start from an empty one. On decompression, sizing the output up front - bulk::decompress, or copy_decode into a Vec::with_capacity - was worth roughly a factor of two in our measurements.
• Streaming data that is already in memory. For a slice you already hold, the one-shot bulk API avoids a copy through the streaming buffers. It is worth about ten percent on decompression, so it is a much smaller effect than sizing the output - the Read/Write wrappers are not the problem people usually assume they are.
• Leaving compression single-threaded. With the zstdmt feature and several workers, compression of a 33 MB input went about three times faster here. It is a loss on small inputs, where there is not enough data to keep the workers busy.

To measure all of these on your own data:

cargo run --release --features zstdmt --example throughput -- <file>

It prints the matching zstd -b<level> command so you can compare against the C library directly.

Cargo features

Enabled by default: legacy, arrays, zdict_builder.

Feature	Default	Description
arrays	yes	Use fixed-size arrays ([u8; N]) as output buffers.
legacy	yes	Decode frames written by zstd versions older than 0.8.
zdict_builder	yes	Train new dictionaries. Using a dictionary always works.
experimental		Expose zstd's experimental API. It has no stability guarantees, and may change between zstd releases.
zstdmt		Multi-threaded compression inside the C library.
thin		Build a smaller C library, at some cost in speed and error reporting.
debug		Enable zstd's debug logs.
no_asm		Do not build the x86-64 assembly.
fat-lto, thin-lto		Cross-language LTO. Only works if clang builds the C library.
doc-cfg		Mark feature-gated items in the generated documentation. Needs a nightly compiler.
wasm		Does nothing; kept so that existing dependants keep building.

The following change how the C library is obtained or built:

Feature	Default	Description
bindgen		Generate the bindings at build time rather than using the pre-generated ones. Needs libclang.
pkg-config		Link against a system-installed libzstd instead of building the bundled source.
vendored		Always build the bundled source, even when pkg-config is also enabled. Useful to force a static build from a dependent crate.
cmake		Build the C library with zstd's own CMake files instead of the cc crate, which handles cross-compilation better. Needs cmake.
rust-backend		Use libzstd-rs-sys, a Rust port of zstd, instead of the C library. Experimental, see below.

The zstd-sys readme also covers the build-time environment variables, and what to do if the linker complains about a hidden zstd symbol being "referenced by DSO".

Compile it yourself

zstd is included as a submodule. To get everything during your clone, use:

git clone https://github.com/gyscos/zstd-rs --recursive

Or, if you cloned it without the --recursive flag, call this from inside the repository:

git submodule update --init

Then, running cargo build should take care of building the C library and linking to it.

Using Rust's global allocator

By default, zstd allocates its internal state (compression windows, match finder tables, stream buffers, dictionaries) with the C runtime's malloc/free, invisible to a custom #[global_allocator].

The with-rust-allocator feature makes every context and dictionary this crate creates (Encoder, Decoder, bulk::Compressor, dictionaries, ...) allocate through Rust's global allocator instead. It is crate-wide: enabling it in your own Cargo.toml also covers zstd usage inside your dependencies, since Cargo unifies features.

zstd = { version = "0.14", features = ["with-rust-allocator"] }

The feature implies experimental on zstd-safe because it relies on zstd's ZSTD_create*_advanced() constructors, which require static linking. It applies to contexts and prepared dictionaries created by the Rust wrappers, including clones. Direct calls to zstd_sys, dictionary training, legacy frame decoding, objects from the seekable format, and explicit zstd thread pools can still allocate through C.

Prepared compression dictionaries use fixed parameters derived from their compression level and dictionary size. Unlike the default C constructor, the custom-allocator constructor does not store the level for later parameter adjustments based on the input size. Compressed output remains interoperable, but enabling this feature can change compression ratio, speed, and memory usage for prepared dictionaries, especially with large inputs and small dictionaries.

Using the Rust port of zstd

The rust-backend feature replaces the C library with libzstd-rs-sys, the Trifecta Tech Foundation's port of zstd to Rust. Nothing else changes: the port exports the same ZSTD_* functions that zstd-sys binds to, and this crate calls them the same way.

zstd = { version = "0.14", features = ["rust-backend"] }

Like the port itself, still a 0.0.1 prerelease, this is experimental:

• The seekable format is not ported. Enabling both features is a compile error.
• The features that configure the C build (thin, debug, no_asm, fat-lto, thin-lto, cmake, pkg-config and vendored) have no effect, and neither does ZSTD_SYS_USE_PKG_CONFIG.
• Multithreading and legacy decoding are always built into the port: zstdmt only controls whether this crate exposes the multithreading API. The port decodes legacy frames from zstd 0.5 to 0.7, whether or not legacy is enabled, but not older ones.
• The port allocates through Rust's global allocator, as with-rust-allocator does for the C library.
• It still needs a C compiler on x86-64 Unix, where the port assembles one file, and it does not build for wasm32-unknown-unknown.
• It needs Rust 1.89 or newer.
• It is slower than the C library in most benchmarks: the pull request that added it has a chart. bench/backends.sh runs the same comparison on your machine.

Build-time bindgen

The bindgen feature requires Rust 1.71 or newer and libclang. The pre-generated bindings remain compatible with Rust 1.64.

This library includes a pre-generated bindings.rs file. You can also generate new bindings at build-time, using the bindgen feature:

cargo build --features bindgen

TODO

• Benchmarks, optimizations, ...

Disclaimer

This implementation is largely inspired by bozaro's lz4-rs.

License

• The zstd C library is under a dual BSD/GPLv2 license. Its decoders for legacy formats (the legacy feature, on by default) also include code under the BSD-2-Clause license.
• This zstd-rs binding library is under a BSD-3-Clause license.