Skip to content
Search

DuckLake

lastmod 2025-05-29created 2025-05-291 edits

Integrated data lake and catalog format

Features

  • A new solution that integrates a data lake and a catalog format
  • Works on top of Parquet files and an SQL database, enabling a simpler data lake implementation than a traditional lakehouse
  • Can manage the metadata catalog on top of various databases such as PostgreSQL, SQLite, MySQL, and DuckDB
  • Supports many features such as snapshots, time travel queries, schema changes, and partitioning, while providing lightweight snapshot handling that does not require frequent compaction
  • Supports a multiplayer DuckDB model in which multiple instances read and write data concurrently, implementing a concurrency model that plain DuckDB does not support
  • DuckLake is a concept covering the specification, the DuckDB extension, and datasets stored in the DuckLake format, and is released under the MIT license

Introducing DuckLake

  • DuckLake is an open format made by the DuckDB team that provides advanced data lake features without a complex lakehouse
  • With just an SQL database and Parquet files, you can build your own data warehouse.
  • Uses a database for metadata management: PostgreSQL, SQLite, MySQL, DuckDB

Key features of DuckLake

  • Data lake operations
    • Snapshots
    • Point-in-time queries (Time travel)
    • Schema evolution
    • Partitioning
  • Lightweight snapshot handling
    • Unlimited number of snapshots can be created
    • Works without frequent compaction
  • ACID transactions
    • Guarantees concurrent access and transactions for multi-table operations
  • Performance-oriented design
    • Uses statistics for filter pushdown
    • Fast queries even on large datasets

Frequently asked questions

  • Why should I use DuckLake?
    • Suitable when you need a lightweight solution that integrates a data lake and a catalog
    • Enables a multiplayer environment where multiple DuckDB instances read and write the same dataset
      • This is a concurrency model not supported by existing DuckDB
    • Even with DuckDB alone you can benefit from point-in-time queries, partitioning, multi-file storage structure, and more
  • What is DuckLake?
    • DuckLake refers to three things:
      • The specification of the DuckLake format
      • The DuckDB extension that supports DuckLake (ducklake extension)
      • The datasets themselves stored in the DuckLake format
  • What is DuckLake's license?
    • Released under the MIT license

See also

Favorite site

author Zer0 sitemap