zfp

llnlBSD-3-Clause活跃维护

把浮点数和整型数组压缩成紧凑格式,同时支持高速随机访问,适合科学计算和数据可视化场景下的大规模数值数据存储与读取。

Compressed numerical arrays that support high-speed random access

1.0.1发布于 2023-12-16 07:47 UTC892star +1 / 7 天184fork同步于 2026-11-10 17:54 UTC

怎么上手

由 README 提炼,执行前请确认
git clone https://github.com/LLNL/zfp.git
cd zfp
mkdir build
cd build
cmake ..
cmake --build . --config Release
ctest
硬件要求按 README 说明

README 未说明具体显存或 GPU 型号要求;支持 Linux、macOS、Windows,CUDA 并行压缩需在 CMake 构建中启用(GNU make 构建不支持 CUDA 等部分特性)。

需要 CUDA
下载:源码 zip·GitHub Releases(安装包)本站不镜像安装包

项目说明

由 README 自动翻译提炼

多维浮点与整数数组的压缩格式,支持高速随机读写单个元素

把浮点数和整型数组压缩成紧凑格式,同时支持高速随机访问,适合科学计算和数据可视化场景下的大规模数值数据存储与读取。

核心能力

  • 提供压缩数组类,支持随机访问单个元素
  • 支持有损但可设误差上限的压缩模式
  • 支持逐位无损压缩模式
  • 支持串行及 OpenMP、CUDA 并行整数组压缩
  • 提供 C、C++ 库及 Python、Fortran 绑定

容易踩的坑

GNU make 构建灵活性较差,不支持 CUDA 等全部特性,需要这些功能时改用 CMake;zfp 对 1D 数组压缩可行但不推荐,主要面向 2D、3D、4D 有空间相关性的数据;完整测试套件需在构建时开启 BUILD_TESTING_FULL 选项。

内容由模型根据 README 生成,可能不完整或不准确,以官方文档为准。看原文 README

最近发版

5 个版本1.0.1
  1. 1.0.1最新正式版2023-12-16 · 2023-12-16 07:47 UTC

    1.0.1 (December 15, 2023)

    This patch release primarily addresses minor bug fixes and is needed to update the zfpy Python wheels.

    Added

    • A new build macro, BUILD_TESTING_FULL, specifies that all unit tests be built; BUILD_TESTING produces a smaller subset of tests. Full tests and documentation are now included in releases.

    Fixed

    • #169: libm dependency is not always correctly detected.
    • #171: ptrdiff_t is not always imported in Cython.
    • #176: cfp API is not exposed via CMake configuration file.
    • #177: Full test suite is not included in release.
    • #181: rpath is not set correctly in executables.
    • #204: Array strides are not passed by value in zFORp.
    • #220: Errors reported with scikit-build when building zfpy.
  2. 1.0.02022-08-01 · 2022-08-01 21:25 UTC
    查看更新说明

    1.0.0 (August 1, 2022)

    This release is not ABI compatible with prior releases due to numerous changes to function signatures and data structures like zfp_field. However, few of the API changes, other than to the cfp C API for compressed arrays, should impact existing code. Note that numerous header files have been renamed or moved relative to prior versions.

    Added

    • zfp::const_array: read-only variable-rate array that supports fixed-precision, fixed-accuracy, and reversible modes.
    • Compressed-array classes for 4D data.
    • const versions of array references, pointers, and iterators.
    • A more complete API for pointers and iterators.
    • cfp support for proxy references and pointers, iterators, and (de)serialization.
    • Support for pointers and iterators into array views.
    • zfp::array::size_bytes() allows querying the size of different components of an array object (e.g., payload, cache, index, metadata, ...).
    • Templated C++ wrappers around the low-level C API.
    • A generic codec for storing blocks of uncompressed scalars in zfp's C++ arrays.
    • Additional functions for querying zfp_field and zfp_stream structs.
    • zfp_config: struct that encapsulates compression mode and parameters.
    • Rounding modes for reducing bias in compression errors.
    • New examples: array, iteratorC, and ppm.

    Changed

    • Headers from array/, cfp/include/, and include/ have been renamed and reorganized into a common include/ directory.
      • The libzfp API is now confined to zfp.h, zfp.hpp, and zfp.mod for C, C++, and Fortran bindings, respectively. These all appear in the top-level include/ directory upon installation.
      • C++ headers now use a .hpp suffix; C headers use a .h suffix.
      • C++ headers like array/zfparray.h have been renamed zfp/array.hpp.
      • C headers like cfp/include/cfparrays.h have been renamed zfp/array.h.
    • size_t and ptrdiff_t replace uint and int for array sizes and strides in the array classes and C/Fortran APIs.
    • zfp_bool replaces int as Boolean type in the C API.
    • bitstream_offset and bitstream_size replace size_t to ensure support for 64-bit offsets into and lengths of bit streams. Consequently, the bitstream API has changed accordingly.
    • All array and view iterators are now random-access iterators.
    • Array inspectors now return const_reference rather than a scalar type like float to allow obtaining a const_pointer to an element of an immutable array.
    • zfp::array::compressed_data() now returns void* instead of uchar*.
    • The array (de)serialization API has been revised, resulting in new zfp::array::header and zfp::exception classes with new exception messages.
    • The array codec class is now responsible for all details regarding compression.
    • The compressed-array C++ implementation has been completely refactored to make it more modular, extensible, and reusable across array types.
    • Array block shapes are now computed on the fly rather than stored.
    • The cfp C API now wraps array objects in structs.
    • The zfpy Python API now supports the more general memoryview over bytes objects for decompression.
    • The zFORp Fortran module name is now zfp instead of zforp_module.
    • Some command-line options for the diffusion example have changed.
    • CMake 3.9 or later is now required for CMake builds.

    Removed

    • zfp::array::get_header() has been replaced with a zfp::array::header constructor that accepts an array object.
    • ZFP_VERSION_RELEASE is no longer defined (use ZFP_VERSION_PATCH).

    Fixed

    • #66: make install overwrites googletest.
    • #84: Incorrect order of parameters in CUDA memset().
    • #86: C++ compiler warns when __STDC_VERSION__ is undefined.
    • #87: CXXFLAGS is misspelled in cfp/src/Makefile.
    • #98: zfp_stream_maximum_size() underestimates size in reversible mode.
    • #99: Incorrect `pr
  3. 0.5.52019-05-06 · 2019-05-06 02:20 UTC
    查看更新说明

    0.5.5 (May 5, 2019)

    • Added support for reversible (lossless) compression of floating-point and integer data.

    • Added methods for serializing and deserializing zfp's compressed arrays.

    • Added Python bindings for compressing NumPy arrays.

    • Added Fortran bindings to zfp's high-level C API.

    • Change:

      • The default compressed-array cache size is now a function of the total number of array elements, irrespective of array shape.
    • Bug fixes:

      • Incorrect handling of execution policy in zfp utility.
      • Incorrect handling of decompression via header in zfp utility.
      • Incorrect cleanup of device memory in CUDA decompress.
      • Tests for failing mallocs.
      • CMake installation of CFP when built.
      • zfp_write_header and zfp_field_metadata now fail if array dimensions are too large to fit in header.
  4. 0.5.42018-10-02 · 2018-10-02 04:56 UTC
    查看更新说明

    0.5.4 (October 1, 2018)

    • Added support for CUDA fixed-rate compression and decompression.

    • Added views into compressed arrays for thread safety, nested array indexing, slicing, and array subsetting.

    • Added C language bindings for compressed arrays.

    • Added support for compressing and decompressing 4D data.

    • Changes:

      • Execution policy now applies to both compression and decompression.
      • Compressed array accessors now return Scalar type instead of const Scalar& to avoid stale references to evicted cache lines.
    • Bug fixes:

      • Handling of negative strides.
      • Command line tool handling of arrays with more than 2^32 elements.
      • bitstream C++ compatibility.
      • Respect minimum cache size request.
  5. 0.5.32018-03-28 · 2018-03-28 19:53 UTC
    查看更新说明

    0.5.3 (March 28, 2018)

    • Added support for OpenMP multithreaded compression (but not decompression).

    • Added options for OpenMP execution to zfp command-line tool.

    • Changed return value of zfp_decompress to indicate the number of compressed bytes processed so far (now returns same value as zfp_compress on success).

    • Added compressed array support for copy construction and assignment via deep copies.

    • Added virtual destructors to enable inheritance from zfp arrays.

用户点评

登录用户的真实使用记录,可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译

    ZFP

    GitHub Actions Test Status Documentation Status Code Coverage R&D100 - Winner

    zfp is a compressed format for representing multidimensional floating-point and integer arrays. zfp provides compressed-array classes that support high throughput read and write random access to individual array elements. zfp also supports serial and parallel (OpenMP and CUDA) compression of whole arrays, e.g., for applications that read and write large data sets to and from disk.

    zfp uses lossy but optionally error-bounded compression to achieve high compression ratios. Bit-for-bit lossless compression is also possible through one of zfp's compression modes. zfp works best for 2D, 3D, and 4D arrays that exhibit spatial correlation, such as continuous fields from physics simulations, natural images, regularly sampled terrain surfaces, etc. zfp compression of 1D arrays is possible but generally discouraged.

    zfp is freely available as open source and is distributed under a BSD license. zfp is primarily written in C and C++ but also includes Python and Fortran bindings. zfp conforms to various language standards, including C89, C99, C11, C++98, C++11, and C++14, and is supported on Linux, macOS, and Windows.

    Quick Start

    To download zfp, type:

    git clone https://github.com/LLNL/zfp.git
    

    zfp may be built using either CMake or GNU make. To use CMake, type:

    cd zfp
    mkdir build
    cd build
    cmake ..
    cmake --build . --config Release
    ctest
    

    This builds the zfp library in the build/lib directory and the zfp command-line executable in the build/bin directory. It then runs the regression tests. The full test suite may be run by enabling the BUILD_TESTING_FULL CMake option during the build step.

    zfp may also be built using GNU make:

    cd zfp
    make
    make test
    

    Note: GNU builds are less flexible and do not support all available features, e.g., CUDA support.

    Finally, zfp may also be installed using package managers such as apt, brew, conda, dnf, pacman, pip, port, spack, vcpkg, and yum. For further installation instructions, please consult the documentation. For examples of how to call the C library and use the C++ array classes, see the examples section.

    Documentation

    Full HTML documentation is available online. A PDF version is also available.

    Further information on the zfp software is included in these files:

    Authors

    zfp was originally developed by Peter Lindstrom at Lawrence Livermore National Laboratory. Please see the Contributors Page for a full list of contributors.

    Citing zfp

    If you use zfp for scholarly research, please cite this paper:

    The algorithm implemented in the current version of zfp is described in the documentation and in the following paper:

    License

    zfp is distributed under the terms of the BSD 3-Clause license. See LICENSE and NOTICE for details.

    SPDX-License-Identifier: BSD-3-Clause

    LLNL-CODE-663824