Delta Lake

lastmod 2025-11-17created 2025-11-171 edits

A storage layer framework that brings reliability to data lakes

See also

• LakeDB
• Data Lakehouse

Favorite site

• 650GB of data (Delta Lake on S3). Polars vs. DuckDB vs. Daft vs. Spark | GeekNews
  • [Original] 650GB of Data (Delta Lake on S3). Polars vs DuckDB vs Daft vs Spark.
  • DuckDB took 16 minutes, Polars 12 minutes, Daft 50 minutes, and PySpark over an hour, confirming that practical processing is possible even on a single node
  • Polars does not support Deletion Vectors; only DuckDB supports the feature, so there is a difference in Lake House compatibility
  • As a result, single-node frameworks proved capable of large-scale data processing even on low-cost hardware, raising the need to reconsider reliance on distributed computing