FastLanes

lastmod 2025-12-30created 2025-12-301 edits

Next-generation big data file format

About

Open-source column-oriented file format designed around modern hardware (SIMD, GPU)
Aims for high-throughput, low-latency data access in analytics and AI workloads
Achieves about 40% better compression ratio than Parquet and up to 40x faster decoding
Introduces a Lane-based layout that minimizes data dependencies, so each unit can be decoded independently
- Achieves extreme data parallelism on SIMD, multi-core CPUs, and GPUs
Designed so auto-vectorization works well even without explicit SIMD code
- Adopts small-batch access that takes CPU and GPU cache characteristics into account
Supports partial decompression, processing data without fully decompressing it, so data engines can run queries directly on compressed data
Exploits correlations between columns through Multi-Column Compression (MCC)
- Provides an expression-based encoding mechanism that addresses the single-column limitation of existing columnar storage formats
Zero-dependency structure that does not rely on external libraries, simplifying builds
- Provides bindings for major languages such as C++, Python, and Rust
Built-in CSV ↔ FastLanes conversion API
- Easy conversion with read_csv() / to_fls()
- Reverse conversion supported with read_fls() / to_csv()
Under development with the goal of integrating with the next-generation data stack, such as GPU decoding and Apache Arrow/DuckDB integration

author Zer0 sitemap