Vortex
lastmod 2024-10-17created 2024-10-171 edits
High-performance Columnar file format
| WARNING |
| Still under active development |
About
- "The LLVM of columnar file formats"
- A Columnar file format that includes a toolkit for handling compressed Apache Arrow arrays in memory, on disk, and over the network
- An ambitious successor to Apache Parquet, supporting 100-200x faster random access reads and 2-10x faster scans while maintaining nearly the same compression ratio and write throughput as Parquet with zstd
- Also supports very wide tables (tens of thousands of columns) and decompression on the GPU
- That is, it features high extensibility, very high speed, and batteries-included functionality
Key features
-
- Logical Types - schema definitions that make no assumptions about the physical layout
- Zero-Copy to Arrow - canonicalized Vortex arrays can be converted to and from Apache Arrow arrays with zero copy
- Extensible Encodings - a pluggable set of physical layouts. In addition to Arrow-compatible encodings, provides state-of-the-art encodings (FastLanes, ALP, FSST, etc.) as extensions
- Cascading Compression - data can be recursively compressed with multiple nested encodings
- Pluggable Compression Strategies - the built-in Compressor is based on BtrBlocks, but other strategies can easily be used
- Compute - basic compute kernels that operate on encoded data (e.g., filter pushdown)
- Statistics - each array carries summary statistics that are optionally computed at read time. Usable by compute kernels and compressors
- Serialization - zero-copy serialization of arrays for IPC and file formats
- Columnar File Format (in progress) - a modern file format that stores compressed array data using the Vortex serde library. Optimized for random access reads and very fast scans. Aims to be the successor to Apache Parquet