GNU make 构建灵活性较差,不支持 CUDA 等全部特性,需要这些功能时改用 CMake;zfp 对 1D 数组压缩可行但不推荐,主要面向 2D、3D、4D 有空间相关性的数据;完整测试套件需在构建时开启 BUILD_TESTING_FULL 选项。
zfp
llnlBSD-3-Clause活跃维护把浮点数和整型数组压缩成紧凑格式,同时支持高速随机访问,适合科学计算和数据可视化场景下的大规模数值数据存储与读取。
Compressed numerical arrays that support high-speed random access
git clone https://github.com/LLNL/zfp.git
cd zfp
mkdir build
cd build
cmake ..
cmake --build . --config Release
ctestREADME 未说明具体显存或 GPU 型号要求;支持 Linux、macOS、Windows,CUDA 并行压缩需在 CMake 构建中启用(GNU make 构建不支持 CUDA 等部分特性)。
多维浮点与整数数组的压缩格式,支持高速随机读写单个元素
把浮点数和整型数组压缩成紧凑格式,同时支持高速随机访问,适合科学计算和数据可视化场景下的大规模数值数据存储与读取。
核心能力
- 提供压缩数组类,支持随机访问单个元素
- 支持有损但可设误差上限的压缩模式
- 支持逐位无损压缩模式
- 支持串行及 OpenMP、CUDA 并行整数组压缩
- 提供 C、C++ 库及 Python、Fortran 绑定
容易踩的坑
- 1.0.1最新正式版2023-12-16 · 2023-12-16 07:47 UTC
1.0.1 (December 15, 2023)
This patch release primarily addresses minor bug fixes and is needed to update the zfpy Python wheels.
Added
- A new build macro,
BUILD_TESTING_FULL, specifies that all unit tests be built;BUILD_TESTINGproduces a smaller subset of tests. Full tests and documentation are now included in releases.
Fixed
- #169:
libmdependency is not always correctly detected. - #171:
ptrdiff_tis not always imported in Cython. - #176: cfp API is not exposed via CMake configuration file.
- #177: Full test suite is not included in release.
- #181:
rpathis not set correctly in executables. - #204: Array strides are not passed by value in zFORp.
- #220: Errors reported with scikit-build when building zfpy.
- A new build macro,
- 1.0.02022-08-01 · 2022-08-01 21:25 UTC
查看更新说明
1.0.0 (August 1, 2022)
This release is not ABI compatible with prior releases due to numerous changes to function signatures and data structures like
zfp_field. However, few of the API changes, other than to the cfp C API for compressed arrays, should impact existing code. Note that numerous header files have been renamed or moved relative to prior versions.Added
zfp::const_array: read-only variable-rate array that supports fixed-precision, fixed-accuracy, and reversible modes.- Compressed-array classes for 4D data.
constversions of array references, pointers, and iterators.- A more complete API for pointers and iterators.
- cfp support for proxy references and pointers, iterators, and (de)serialization.
- Support for pointers and iterators into array views.
zfp::array::size_bytes()allows querying the size of different components of an array object (e.g., payload, cache, index, metadata, ...).- Templated C++ wrappers around the low-level C API.
- A generic codec for storing blocks of uncompressed scalars in zfp's C++ arrays.
- Additional functions for querying
zfp_fieldandzfp_streamstructs. zfp_config: struct that encapsulates compression mode and parameters.- Rounding modes for reducing bias in compression errors.
- New examples:
array,iteratorC, andppm.
Changed
- Headers from
array/,cfp/include/, andinclude/have been renamed and reorganized into a commoninclude/directory.- The libzfp API is now confined to
zfp.h,zfp.hpp, andzfp.modfor C, C++, and Fortran bindings, respectively. These all appear in the top-levelinclude/directory upon installation. - C++ headers now use a
.hppsuffix; C headers use a.hsuffix. - C++ headers like
array/zfparray.hhave been renamedzfp/array.hpp. - C headers like
cfp/include/cfparrays.hhave been renamedzfp/array.h.
- The libzfp API is now confined to
size_tandptrdiff_treplaceuintandintfor array sizes and strides in the array classes and C/Fortran APIs.zfp_boolreplacesintas Boolean type in the C API.bitstream_offsetandbitstream_sizereplacesize_tto ensure support for 64-bit offsets into and lengths of bit streams. Consequently, thebitstreamAPI has changed accordingly.- All array and view iterators are now random-access iterators.
- Array inspectors now return
const_referencerather than a scalar type likefloatto allow obtaining aconst_pointerto an element of an immutable array. zfp::array::compressed_data()now returnsvoid*instead ofuchar*.- The array (de)serialization API has been revised, resulting in new
zfp::array::headerandzfp::exceptionclasses with new exception messages. - The array
codecclass is now responsible for all details regarding compression. - The compressed-array C++ implementation has been completely refactored to make it more modular, extensible, and reusable across array types.
- Array block shapes are now computed on the fly rather than stored.
- The cfp C API now wraps array objects in structs.
- The zfpy Python API now supports the more general
memoryviewoverbytesobjects for decompression. - The zFORp Fortran module name is now
zfpinstead ofzforp_module. - Some command-line options for the
diffusionexample have changed. - CMake 3.9 or later is now required for CMake builds.
Removed
zfp::array::get_header()has been replaced with azfp::array::headerconstructor that accepts an array object.ZFP_VERSION_RELEASEis no longer defined (useZFP_VERSION_PATCH).
Fixed
- #66:
make installoverwrites googletest. - #84: Incorrect order of parameters in CUDA
memset(). - #86: C++ compiler warns when
__STDC_VERSION__is undefined. - #87:
CXXFLAGSis misspelled incfp/src/Makefile. - #98:
zfp_stream_maximum_size()underestimates size in reversible mode. - #99: Incorrect `pr
- 0.5.52019-05-06 · 2019-05-06 02:20 UTC
查看更新说明
0.5.5 (May 5, 2019)
-
Added support for reversible (lossless) compression of floating-point and integer data.
-
Added methods for serializing and deserializing zfp's compressed arrays.
-
Added Python bindings for compressing NumPy arrays.
-
Added Fortran bindings to zfp's high-level C API.
-
Change:
- The default compressed-array cache size is now a function of the total number of array elements, irrespective of array shape.
-
Bug fixes:
- Incorrect handling of execution policy in zfp utility.
- Incorrect handling of decompression via header in zfp utility.
- Incorrect cleanup of device memory in CUDA decompress.
- Tests for failing mallocs.
- CMake installation of CFP when built.
- zfp_write_header and zfp_field_metadata now fail if array dimensions are too large to fit in header.
-
- 0.5.42018-10-02 · 2018-10-02 04:56 UTC
查看更新说明
0.5.4 (October 1, 2018)
-
Added support for CUDA fixed-rate compression and decompression.
-
Added views into compressed arrays for thread safety, nested array indexing, slicing, and array subsetting.
-
Added C language bindings for compressed arrays.
-
Added support for compressing and decompressing 4D data.
-
Changes:
- Execution policy now applies to both compression and decompression.
- Compressed array accessors now return Scalar type instead of const Scalar& to avoid stale references to evicted cache lines.
-
Bug fixes:
- Handling of negative strides.
- Command line tool handling of arrays with more than 2^32 elements.
- bitstream C++ compatibility.
- Respect minimum cache size request.
-
- 0.5.32018-03-28 · 2018-03-28 19:53 UTC
查看更新说明
0.5.3 (March 28, 2018)
-
Added support for OpenMP multithreaded compression (but not decompression).
-
Added options for OpenMP execution to zfp command-line tool.
-
Changed return value of zfp_decompress to indicate the number of compressed bytes processed so far (now returns same value as zfp_compress on success).
-
Added compressed array support for copy construction and assignment via deep copies.
-
Added virtual destructors to enable inheritance from zfp arrays.
-
用户点评
登录用户的真实使用记录,可按版本和硬件筛选还没有人点评。第一个说出真实体验的人最有价值。
ZFP
zfp is a compressed format for representing multidimensional floating-point and integer arrays. zfp provides compressed-array classes that support high throughput read and write random access to individual array elements. zfp also supports serial and parallel (OpenMP and CUDA) compression of whole arrays, e.g., for applications that read and write large data sets to and from disk.
zfp uses lossy but optionally error-bounded compression to achieve high compression ratios. Bit-for-bit lossless compression is also possible through one of zfp's compression modes. zfp works best for 2D, 3D, and 4D arrays that exhibit spatial correlation, such as continuous fields from physics simulations, natural images, regularly sampled terrain surfaces, etc. zfp compression of 1D arrays is possible but generally discouraged.
zfp is freely available as open source and is distributed under a BSD license. zfp is primarily written in C and C++ but also includes Python and Fortran bindings. zfp conforms to various language standards, including C89, C99, C11, C++98, C++11, and C++14, and is supported on Linux, macOS, and Windows.
Quick Start
To download zfp, type:
git clone https://github.com/LLNL/zfp.git
zfp may be built using either CMake or GNU make. To use CMake, type:
cd zfp
mkdir build
cd build
cmake ..
cmake --build . --config Release
ctest
This builds the zfp library in the build/lib directory and the zfp
command-line executable in the build/bin directory. It then runs
the regression tests. The full test suite may be run by enabling the
BUILD_TESTING_FULL CMake option during the build step.
zfp may also be built using GNU make:
cd zfp
make
make test
Note: GNU builds are less flexible and do not support all available features, e.g., CUDA support.
Finally, zfp may also be installed using package managers such as apt,
brew, conda, dnf, pacman, pip, port, spack, vcpkg, and
yum. For further installation instructions, please consult the
documentation.
For examples of how to call the C library and use the C++ array classes,
see the examples
section.
Documentation
Full HTML documentation is available online. A PDF version is also available.
Further information on the zfp software is included in these files:
- Change log: see CHANGELOG.md.
- Support and additional resources: see SUPPORT.md.
- Code contributions: see CONTRIBUTING.md.
Authors
zfp was originally developed by Peter Lindstrom at Lawrence Livermore National Laboratory. Please see the Contributors Page for a full list of contributors.
Citing zfp
If you use zfp for scholarly research, please cite this paper:
- Peter Lindstrom. Fixed-Rate Compressed Floating-Point Arrays. IEEE Transactions on Visualization and Computer Graphics, 20(12):2674-2683, December 2014. doi:10.1109/TVCG.2014.2346458.
The algorithm implemented in the current version of zfp is described in the documentation and in the following paper:
- James Diffenderfer, Alyson Fox, Jeffrey Hittinger, Geoffrey Sanders, Peter Lindstrom. Error Analysis of ZFP Compression for Floating-Point Data. SIAM Journal on Scientific Computing, 41(3):A1867-A1898, June 2019. doi:10.1137/18M1168832.
License
zfp is distributed under the terms of the BSD 3-Clause license. See LICENSE and NOTICE for details.
SPDX-License-Identifier: BSD-3-Clause
LLNL-CODE-663824