vortex

vortex-dataApache-2.0活跃维护

一套可扩展的列柱式压缩框架，也是目前最快的开源列柱式文件格式，面向多模态和 Arrow 数据的高效存储与读取。

An extensible, state-of-the-art framework for columnar compression, and the fastest FOSS columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.

0.93.0发布于 2026-11-06 14:49 UTC3,312star +2 / 7 天258fork同步于 2026-11-10 05:40 UTC

列柱式压缩文件格式多模态数据 Arrow Rust

对比 GitHub 官网 array arrow compression file multimodal

怎么上手

由 README 提炼，执行前请确认

原文

cargo add vortex
uv add vortex-data

硬件要求按 README 说明

README 未说明显存或 GPU 要求，仅提及可选用 MiMalloc 提升性能

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼

没人工点评？写下你的经验

新一代列式文件格式与工具包，为对象存储上的数据系统提供比 Parquet 更快的读写

一套可扩展的列柱式压缩框架，也是目前最快的开源列柱式文件格式，面向多模态和 Arrow 数据的高效存储与读取。

核心能力

随机访问读取比 Parquet 快 100 倍
扫描快 10-20 倍、写入快 5 倍，压缩率相近
可插拔编码、类型系统、压缩与布局策略
与 Apache Arrow 零拷贝互操作
提供 Arrow、DataFusion、DuckDB、Spark、Pandas、Polars 集成

容易踩的坑

库 API 可能随版本变化，只有 0.36.0 起的文件格式保证向后兼容；开发构建需先执行 git submodule update --init --recursive 并 uv sync --all-packages；vx 命令行工具需单独安装 vortex-tui 包

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本0.93.0

全部 release

0.93.0最新正式版2026-11-06 · 2026-11-06 14:49 UTC
Changes

✨ Features
- feat(cuda): expose the existing memory pool through FFI (#11209) @0ax1
- feat(spark): let VortexOutputWriter store footer metadata (#11137) @ChristopherWirt
- Property-based tests for count_ones (#11280) @myrrc
🚀 Performance
- perf(cuda): use 8Mi-row blocks by default (#11212) @0ax1
- perf(cuda): write-combine pinned staging buffers (#11210) @0ax1
- Push down array_transform in DataFusion (#11031) @mhk197
🐛 Bug Fixes
- Return UUID column statistics to Java as bytes (#11299) @robert3005
- Give each inline V2 scan root its own IO session (#11226) @joseph-isaacs
- fix(cuda): honor requested file-read alignment (#11291) @0ax1
- fix(cuda): release file mappings after their own copies complete (#11290) @0ax1
- fix(cuda): tie file reads to the opened handle (#11289) @0ax1
- fix(cuda): stage reads when file mapping is unavailable (#11288) @0ax1
🧰 Maintenance
- Benchmark lowercase label equality in PolarSignals (#11295) @asubiotto
0.92.02026-11-06 · 2026-11-06 10:45 UTC
查看更新说明
Changes

⚠️ Breaks
- Read legacy vortex.decimal nodes into the canonical array or a Widen (#10995) @connortsui20
✨ Features
- Round-trip extension types without an Arrow plugin through Arrow (#11224) @connortsui20
- Add LLDB formatters and Scheme pretty-printers (#11253) @jiengup
- PartitionSession: register partition rules dynamically (#10812) @myrrc
- perf(jni): bound completed scan outputs with background prefetch (#11198) @ChristopherWirt
🚀 Performance
- perf(cuda): share the default context across live sessions (#11263) @0ax1
- perf(onpair): gather sparse token ranges before decoding (#10370) @joseph-isaacs
- duckdb: parallelize flush stage of COPY TO (#11269) @myrrc
- perf(dict): reuse validated indices during string gathers (#10369) @joseph-isaacs
- perf(cuda): retain freed memory in context-owned pools (#11262) @0ax1
- perf(arrow): preserve encoding-specific byte export builders (#10368) @joseph-isaacs
- perf(dict): append decoded children without expanding rows (#10367) @joseph-isaacs
- Export sparse dictionary batches as plain values in DuckDB (#11255) @robert3005
- Plan and execute common RowFn batches in one dispatch (#10675) @connortsui20
- Push down array_sum in DataFusion (#10854) @mhk197
- Speed up RunEnd filter with a rank directory and POPCNT dispatch (#10575) @robert3005
- Push DataFusion dynamic filters into V2 scans (#11215) @joseph-isaacs
🐛 Bug Fixes
- Scan extension columns over list, map, and string storage in DataFusion (#11225) @connortsui20
- Export DuckDB chunked arrays across chunk boundaries (#11204) @robert3005
- Trim child data in Arrow map and run-end imports (#10381) @palaska
- Fix the return dtype of grouped is_sorted and is_constant (#11238) @connortsui20
📖 Documentation
- Docs and Explorer: the footer and the docs' 404 page match vortex.dev (#11266) @lwwmanning
- Show X, Slack, and the star count in the docs' and Explorer's header, and move the theme switch to the footer (#11251) @lwwmanning
- Show the same header, footer, 404, and theme switch on the docs and Explorer (#11249) @lwwmanning
- docs: require vortex_ensure_eq for equality checks (#11246) @connortsui20
- Docs: draw the docs with the Vortex design system (#11207) @lwwmanning
🧰 Maintenance
- refactor(cuda): remove unused driver API (#10879) @0ax1
- refactor(cuda): simplify benchmark session setup (#10885) (#10885) @0ax1
- refactor(cuda): return operation errors instead of deferring them (#11264) @0ax1
- Add the nested TPC-H benchmark (tpch-nested) to vortex-bench (#11191) @mprammer
- bench: add statpopgen compression coverage (#11043) @danking
- refactor(cuda): organize driver APIs by resource (#11261) @0ax1
- Fix the mixed-dimension chunk test in vortex-bench (#11236) @connortsui20
0.91.02026-11-04 · 2026-11-04 17:51 UTC
查看更新说明
Changes

⚠️ Breaks
- Add shared conversion to canonical decimal arrays or Widen (#11005) @connortsui20
- Reject nullable nested levels in list-based spatial types (#11183) @connortsui20
- Preserve extension literals in DataFusion comparisons (#11067) @connortsui20
- Require matching dtypes for extension comparisons (#11016) @connortsui20
- Change extension dtype nullability without validating again (#11012) @connortsui20
- Separate the read and write IDs of array plugins (#11057) @connortsui20
- Vortex spark integration is a fileformat not a tableprovider (#9658) @robert3005
- Return errors for invalid temporal types and values (#11015) @connortsui20
- DictStrategy probes a fixed row sample instead of the first chunk (#10927) @mhk197
- Layout plugins read and write more than one wire ID (#10704) @mhk197
- Make uncompressed_size_in_bytes always exact and drop its options (#10938) @joseph-isaacs
- Add a canonical decimal width and import decimals at it (#10897) @connortsui20
- Move RunEnd into vortex-array (#10703) @mhk197
- Make protobuf metadata fields crate-private (#10831) @connortsui20
- Decode stored DecimalType and PType values strictly (#10755) @connortsui20
- Return a borrowed scalar from as_constant (#10790) @robert3005
- Simplify error types, remove kind from struct (#10080) @robert3005
- perf(cuda): migrate consumers to vortex-cuda-driver (#10722) @0ax1
✨ Features
- feat(array): migrate simple encodings to scalar probes (#10859) @jiengup
- Explorer: draw it with the Vortex design system, and build it with Bun (#11193) @lwwmanning
- Property-based tests for Mask (#11167) @myrrc
- Replace layout lowering with a VTable::new_plan method (#11200) @joseph-isaacs
- Expose the existing footer segment end offset (#11196) @brancz
- test: remove alp tests which are covered by other tests (#11155) @myrrc
- test: remove vortex-duckdb tests which are covered by other tests (#11153) @myrrc
- Support decimal values in Widen (#10946) @connortsui20
- Add ScalarFnVTable::execute_step for iterative scalar fn execution (#11100) @joseph-isaacs
- Read Vortex file column statistics through vortex-jni (#10863) @robert3005
- Select editions from JNI and Python, and write files for readers before vortex.zoned (#10805) @robert3005
- fix(python): read pyarrow files positionally in open_readable (#11074) @jackylee-ch
- feat(spark): push nested struct pruning down to the native scan (#10957) @ChristopherWirt
- Add serialization for aggregate functions (#11048) @gatesn
- Record the uncompressed size in file stats when editions allow it (#10923) @joseph-isaacs
- Add uncompressed_size_in_bytes to the preview edition (#10922) @joseph-isaacs
- Execute interleaved decimal arrays (#10940) @connortsui20
- Add a scan for the range of valid decimal values (#10939) @connortsui20
- Add an in-memory Widen encoding for integers (#10864) @connortsui20
- Add a naive Unary encoding in a new vortex-elias-fano crate (#10609) @rapour
- duckdb: support windows-x86 and windows-arm64 (#9570) @myrrc
- string-bench: benchmark onpair-16 and add the OnPair paper datasets (#10920) @joseph-isaacs
- Add for_each_unset_index to BitBuffer (#10866) @rapour
- Push DataFusion dynamic filters through Scan API expression trees (#10895) @gatesn
- Skip file filters proven true until their dynamic producer changes (#10894) @gatesn
- Derive live statistics proofs for dynamic expressions (#10869) @gatesn
- Add generic dynamic expressions with generation tracking (#10865) @gatesn
- feat(fsst): Compute uncompressed size without decoding (#9755) @lwwmanning
- Add list_transform (#10503) @mhk197
- Rebuild lists over only the elements valid lists reference (#10743) @mhk197
- Expose aggregate statistics via Scan API (#10838) @gatesn
- Add an unstable edition that includes all unreleased and upcoming encodings (#10558) @robert3005
- Drive V2 scans in the DuckDB extension with a custom driver (#10809) @joseph-isaacs
0.90.02026-10-23 · 2026-10-23 13:05 UTC
查看更新说明
Changes

⚠️ Breaks
- Remove legacy_session and the deprecated APIs that rely on it (#10595) @robert3005
- Validate List and ListView arrays with an explicit execution context (#10594) @robert3005
- Validate VarBin arrays with an explicit execution context (#10593) @robert3005
- Thread execution contexts through Patches (#10592) @robert3005
✨ Features
- Add select_unset to BitBuffer (#10666) @rapour
- Poc index layout (#9706) @thorfour
- Support List/Struct/Map scalars in DuckDB conversion (#10639) @robert3005
- Prune in_list expressions (#10617) @robert3005
- feat(aggregate): register is_sorted and min_max for temporal and UUID extension types (#10616) @joseph-isaacs
🚀 Performance
- perf(sequence): take through the fallible lane kernels (#10142) @robert3005
- Decode DateTimeParts values in single pass (#10140) @robert3005
- Track kept patches in a bitmap when filtering patches (#10642) @robert3005
- feat(scan): cache V2 scan state across scans of one file (#10631) @joseph-isaacs
🐛 Bug Fixes
- Return an empty array from fill_null on an empty input (#10627) @ump45nose
- Drop an empty ListView's elements when rebuilding (#10655) @mhk197
- Preserve extension metadata on Arrow dictionary import (#10645) @asubiotto
- Push down comparisons through REE-to-dictionary casts (#10637) @yiquanzhou
- fix(ipc): fill the buffer on partial reads in the sync message reader (#10284) @jackylee-ch
🧰 Maintenance
- duckdb: replace hash to 2.0.0 (#10709) @myrrc
- Refactor Interleave to use named child slots (#10688) @jiengup
- Split TPC-H and TPC-DS benchmark tables into ~200MB files (#10519) @joseph-isaacs
- Update dependency storybook to v10.6.2 [SECURITY] (#10682) @renovate[bot]
- ci: Alpine 3.24 musl container and move small-runner jobs to amd64-ci-medium (#10667) @joseph-isaacs
- Use same tpchgen-rs for sqllogic test and bench (#10656) @robert3005
- random-access-bench: read only the selected rows from Parquet (#10647) @joseph-isaacs
- random-access-bench: take Lance rows from its single fragment (#10649) @joseph-isaacs
- Update webpki packages to resolve security warnings (#10662) @robert3005
- Update dependency vite to v8.3.3 [SECURITY] (#10654) @renovate[bot]
- Add Ashvin Agrawal and Matthew Katz as Committers (#10644) @mhk197
- file_segment_source_panic_propagates_to_all_concurrent_readers test no longer depends on time (#10636) @robert3005
- Fix random fail in runtime::tokio::tests::test_spawn_and_abort (#10632) @robert3005
- Drop VarBinView's debug validation in new_unchecked (#10584) @robert3005
- Validate FSST codes bounds only where an execution context exists (#10588) @robert3005
0.89.02026-10-20 · 2026-10-20 11:00 UTC
查看更新说明
Changes

⚠️ Breaks
- C++ FFI field access method doesn't require a session (#10611) @robert3005
- Pass an execution context to array value display (#10583) @robert3005
- Take a session in vx_array_get_field and vx_array_invalid_count (#10589) @robert3005
- duplicate ZstdBuffer's validity into a slot instead of using the legacy session (#10586) @robert3005
- Take an execution context in CanonicalCudaExt::into_host (#10590) @robert3005
- Blocked FoR: store a global reference scalar as metadata rather than child (#10222) @mhk197
- Move Shared array out of vortex-array into vortex-layout (#10495) @robert3005
✨ Features
- Convert FixedSizeList scalars to and from DuckDB ARRAY values (#10629) @robert3005
- feat(aggregate): add ExtensionStoragePushdown to push an aggregate into an extension type's storage (#10566) @joseph-isaacs
- Expose list membership with SQL null semantics in Java and Python (#10053) @robert3005
- Duckdb 2.0 (#9141) @myrrc
- feat(array): enable CaseWhen expression serialization (#10604) @blaginin
- feat(sparse): push LIKE through Sparse arrays (#10538) @robert3005
- feat(layout): zoned pruning, shares, and dictionary value reuse in physical plans (#10462) @joseph-isaacs
- perf(python): release the GIL once per scan export (#10497) @robert3005
- feat(python): open local files as memory maps (#10498) @robert3005
- Optimize list_contains with prepared constant sets (#10052) @robert3005
- feat: expose Spark scan schema and pushed predicates (#9956) @xiaoh1024
- Blocked BitPacked: Add Compressor support via BitPackingScheme mode (#10245) @mhk197
- Support Polars 2 (#10398) @robert3005
- feat(layout): compile evals, row indices, and takes to pipelines (#10393) @joseph-isaacs
- feat(layout): compile segment scans, filters, concatenations, and packs to pipelines (#10392) @joseph-isaacs
- feat(layout): add push-based pipelines and the scan that runs them (#10391) @joseph-isaacs
- Support filters with limit in ScanBuilder (#8766) @AdamGS
- Add more scalar literal types to the Python and JNI bindings (#10079) @robert3005
- feat(array): support fill null for UTF-8 and binary arrays (#10044) @1fanwang
- feat(python): support row indices in VortexFile.to_arrow (#10373) @joseph-isaacs
🚀 Performance
- Plan V2 random access from known row selections (#10625) @joseph-isaacs
- perf(fastlanes): answer BitPacked compare and between from the bit width, between through the fused kernel (#10559) @mightsleep
- perf(array): snapshot aggregate kernels in ExecutionCtx (#10561) @joseph-isaacs
- Decode short RunEnd runs with fixed-size block writes (#10488) @joseph-isaacs
- Size the list layout split grid by row count (#10531) @mhk197
- Preserve contiguous index ranges in BitPacked take (#10520) @blaginin
- perf(fsst): gather sparse takes from small arrays with large payloads (#10509) @blaginin
- Deduplicate execute is_done check with done_canonical check (#10387) @robert3005
- perf(datafusion): avoid Arrow session lookup for ordinary scalars (#10541) @joseph-isaacs
- perf: make FieldSelection normalization linear (#10517) @joseph-isaacs
- perf: read pass-through struct projections directly (#10515) @joseph-isaacs
- perf: move decoded stats into arrays without stats (#10516) @joseph-isaacs
- Optimise bitpacked filtering and take for valid runs (#10281) @robert3005
- fix(datetime-parts): keep null rows null and order by parts (#10145) @robert3005
- perf: exit compute_referenced_values_mask early once every value is referenced (#10423) @joseph-isaacs
- Decode nullable RunEnd validity by clearing null streaks (#10487) @joseph-isaacs
- Drop DuckDB file scan state outside the multi-file lock (#10486) @robert3005
- perf(layout): skip chunks without selected rows (#10491) @robert3005
- Bound DashMap shard counts for per-reader caches (#10485) @robert3005
- Blocked BitPacked: fuse FoR decode over blocked children (#10251) @mhk197
- perf(mask): read intersect_by_ran

用户点评

登录用户的真实使用记录，可按版本和硬件筛选

共 0 条

还没有人点评。第一个说出真实体验的人最有价值。

README 原文

未翻译

在 GitHub 打开

🌪️ Vortex

Join the community on Slack! | Documentation | Performance Benchmarks

If you are interested in closer collaboration, please email [email protected]

Overview

Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for building data systems backed by object storage. It provides:

Blazing Fast Performance
- 100x faster random access reads (vs. modern Apache Parquet)
- 10-20x faster scans
- 5x faster writes
- Similar compression ratios
- Efficient support for wide tables with zero-copy/zero-parse metadata
Extensible Architecture
- Modeled after Apache DataFusion's extensible approach
- Pluggable encoding system, type system, compression strategy, & layout strategy
- Zero-copy compatibility with Apache Arrow
Open Source, Neutral Governance
- A Linux Foundation (LF AI & Data) Project
- Apache-2.0 Licensed
Integrations
- Arrow, DataFusion, DuckDB, Spark, Pandas, Polars, & more
- Apache Iceberg (coming soon)

🟢 Development Status: Library APIs may change from version to version, but we now consider the file format stable. From release 0.36.0, all future releases of Vortex should maintain backwards compatibility of the file format (i.e., be able to read files written by any earlier version >= 0.36.0).

Key Features

Core Capabilities

Logical Types - Clean separation between logical schema and physical layout
Zero-Copy Arrow Integration - Seamless conversion to/from Apache Arrow arrays
Extensible Encodings - Pluggable physical layouts with built-in optimizations
Cascading Compression - Support for nested encoding schemes
High-Performance Computing - Optimized compute kernels for encoded data
Rich Statistics - Lazy-loaded summary statistics for optimization

Technical Architecture

Logical vs Physical Design

Vortex strictly separates logical and physical concerns:

Logical Layer: Defines data types and schema
Physical Layer: Handles encoding and storage implementation
Built-in Encodings: Compatible with Apache Arrow's memory format
Extension Encodings: Optimized compression schemes (RLE, dictionary, etc.)

Quick Start

Installation

Rust Crate

All features are exported through the main vortex crate.

cargo add vortex

Python Package

uv add vortex-data

Command Line UI (vx)

For browsing the structure of Vortex files, you can use the vx command-line tool.

# Install pre-built binary (fast, recommended)
cargo binstall vortex-tui

# Or build from source
cargo install vortex-tui --locked

# Or run via Python without installing
uvx --from vortex-data vx --help

# Usage
vx browse <file>

Development Setup

Prerequisites (macOS)

# Optional but recommended dependencies
brew install flatbuffers protobuf  # For .fbs and .proto files
brew install duckdb               # For benchmarks

# Install Rust toolchain
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# or
brew install rustup

# Initialize submodules
git submodule update --init --recursive

# Setup dependencies with uv
uv sync --all-packages

See the development workflows for Python binding and documentation development, including Maturin rebuilds, targeted tests, and documentation checks.

rust-toolchain.toml pins the toolchain used for development and CI, and is kept on the latest stable release. Building Vortex as a dependency only requires a toolchain that satisfies the Rust version compatibility policy.

Benchmarking

Use vx-bench to run benchmarks comparing engines (DataFusion, DuckDB) and formats (Parquet, Vortex):

# Install the benchmark orchestrator
uv tool install "bench_orchestrator @ ./bench-orchestrator/"

# Run TPC-H benchmarks
vx-bench run tpch --engine datafusion,duckdb --format parquet,vortex

# Compare results
vx-bench compare --run latest

See bench-orchestrator/README.md for full documentation.

Performance Optimization

For optimal performance, we suggest using MiMalloc:

#[global_allocator]
static GLOBAL_ALLOC: MiMalloc = MiMalloc;

Project Information

Rust Version Compatibility Policy

The policy: Vortex supports the four most recent stable minor releases. Writing the latest stable release as 1.N, that means 1.N, 1.N-1, 1.N-2, and 1.N-3 all build Vortex, so the three minor releases older than the latest stable release. Only the minor version is constrained; Minimum Supported Rust Version (MSRV) declared in Cargo.toml must be no newer than 1.N-3, patch releases are never a factor.

An MSRV older than 1.N-3 is always acceptable — supporting extra releases cannot break the guarantee. An MSRV newer than 1.N-3 does not meet the policy. For example, once 1.98 is the latest stable release:

Declared MSRV	Status
`1.94` or older	Acceptable — supports more releases than required
`1.95`	Exactly on policy
`1.96` or newer	Does not meet the policy

The MSRV is raised in occasional deliberate steps rather than on every Rust release, so it drifts relative to that bound.

How the policy is applied:

The MSRV is declared once, as rust-version in the root Cargo.toml, and inherited by every crate in the workspace. That value, not this document, is the source of truth.
The toolchain pinned in rust-toolchain.toml tracks the latest stable release and is independent of the MSRV. It is what contributors and most CI jobs build with.
CI enforces the declared MSRV in the Rust (MSRV) job, which builds the publishable crates with exactly that toolchain. When it fails, the first choices are to express the code without the newer Rust feature, or to hold back the dependency update that raised the requirement. Raising rust-version is a last resort.

License

Licensed under the Apache License, Version 2.0.

Governance

Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is a sub-project of the Linux Foundation Projects. The governance model is documented in CONTRIBUTING.md and is subject to the terms of the Technical Charter.

Contributing

Please do read CONTRIBUTING.md before you contribute.

Reporting Vulnerabilities

If you discover a security vulnerability, please email [email protected]. See SECURITY.md for our full security policy, including emergency reporting and CRA stewardship.

Trademarks

Acknowledgments

The Vortex project benefits enormously from groundbreaking work from the academic & open-source communities.

Research in Vortex

BtrBlocks - Efficient columnar compression
FastLanes & FastLanes on GPU - High-performance integer compression
FSST - Fast random access string compression
ALP & G-ALP - Adaptive lossless floating-point compression
Procella - YouTube's unified data system
Anyblob - High-performance access to object storage
ClickHouse - Fast analytics for everyone
MonetDB/X100 - Hyper-Pipelining Query Execution
Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Format for the Many-Core Age
The FastLanes File Format - Expression Operators

Vortex in Research

Anyblox - A Framework for Self-Decoding Datasets
F3 - Open-Source Data File Format for the Future

怎么上手

项目说明

核心能力

容易踩的坑

最近发版

Changes

✨ Features

🚀 Performance

🐛 Bug Fixes

🧰 Maintenance

Changes

⚠️ Breaks

✨ Features

🚀 Performance

🐛 Bug Fixes

📖 Documentation

🧰 Maintenance

Changes

⚠️ Breaks

✨ Features

Changes

⚠️ Breaks

✨ Features

🚀 Performance

🐛 Bug Fixes

🧰 Maintenance

Changes

⚠️ Breaks

✨ Features

🚀 Performance

用户点评

README 原文

🌪️ Vortex

Overview

Key Features

Core Capabilities

Technical Architecture

Logical vs Physical Design

Quick Start

Installation

Rust Crate

Python Package

Command Line UI (vx)

Development Setup

Prerequisites (macOS)

Benchmarking

Performance Optimization

Project Information

Rust Version Compatibility Policy

License

Governance

Contributing

Reporting Vulnerabilities

Trademarks

Acknowledgments

Research in Vortex

Vortex in Research

Open Source Inspiration

Thanks to all contributors who have shared their knowledge and code with the community! 🚀

相关项目