Lance
lastmod 2023-06-08created 2023-06-082 edits
Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, PyArrow, with more integrations coming..
A modern columnar data format optimized for ML workflows
Features
- Converts from Parquet in 2 lines of code; supports 100x faster random access, vector index/search, data versioning, etc.
- Compatible with pandas, duckdb, polars, pyarrow, etc.
- Zero-copy, automatic versioning
- Best suited for
- Building search engines and feature stores
- Training large ML models that need high-performance IO and shuffles
- Storing/querying/inspecting complex nested data such as robotics data and large blobs (images, point clouds, ...)