LakeDB
lastmod 2025-11-17created 2025-02-023 edits
Envisioning LakeDB - the next evolution of the LakeHouse architecture
About
- The data management field is changing rapidly, and the Data Lakehouse concept is emerging due to growing demand for cloud storage and real-time analytics
- Projects such as Apache Iceberg, Apache Hudi, and Delta Lake have provided core features to the Lakehouse architecture, such as schema evolution, ACID transactions, and efficient updates
- Google's internal system Napa presents an approach that goes a step beyond current Lakehouse solutions
- Going further, adding ideas from other systems such as Apache Pinot, proposes LakeDB as the next form of the Lakehouse
The current Lakehouse landscape: Iceberg, Hudi, Delta Lake
- Apache Iceberg
- Supports schema evolution, time travel, and efficient query planning through sophisticated metadata management
- Provides consistency guarantees for large analytical datasets
- Uses log-structured storage and indexing to handle upserts, deletes, and CDC (change data capture) efficiently
- Emphasizes data mutation and incremental processing
- Provides ACID transactions for Spark and big data workloads
- Supports schema validation, time travel, and unified batch and streaming processing
- Provides some declarative pipelines and materialized views through Delta Live Tables
Google's Napa: a holistic approach
- A complete analytical data management system that supports low-latency queries and continuous data ingestion over large-scale data
- LSM (Log-Structured Merge)-Tree-based ingestion
- Adopts the LSM-tree approach for high write throughput, with a structure that merges with existing data through compaction
- Automatically maintains and updates materialized views to accelerate queries
- Provides consistent point-in-time management across the system
- Allows fine-grained control over the balance between data freshness and query performance
- Uses Google's F1 Query engine to process complex analytical queries efficiently
- Data freshness, performance, cost, and more can be tuned to user needs
Comparing Napa, Iceberg, Hudi, and Delta Lake
- Napa is a holistic analytical data management system that supports fast queries through LSM-based materialized views and finely controls data freshness with a technique called Queryable Timestamp (QT). It is useful when you want to analyze and report on large-scale data quickly, and its high configuration flexibility lets you balance cost, performance, and freshness appropriately.
- Iceberg is a project focused on the table format; it manages large data files through metadata and provides features such as atomic updates. It is mainly used in data warehousing or in data lake environments that need features such as schema evolution and time travel, and its configuration options are mostly focused on table layout and metadata.
- Hudi is a platform that brings database-like features to the data lake; it efficiently handles data mutation operations such as upserts and deletes through log-structured storage and indexing techniques. It is designed to handle real-time ingestion, change data capture (CDC), and regulatory compliance requirements such as GDPR well, but does not provide materialized views by default.
- Delta Lake is a storage layer that supports ACID transactions; it handles batch and streaming jobs in a unified way and also provides schema enforcement and time travel. To improve query performance it integrates with Spark to use data skipping and caching, and it also supports concepts like materialized views through the separate Delta Live Tables. It is often used when you want to add reliability and transactional features to various data lake environments.
The case for LakeDB: inspired by Napa, learning from elsewhere
- Combines innovative ideas from Napa, Apache Pinot, and others to propose a more unified and flexible concept called LakeDB
- LakeDB aims to be a data management solution where users declaratively express their requirements (freshness, cost, consistency, index usage, etc.) and the system optimizes on its own
- Unified storage, ingestion, metadata, and query processing
- Storage, ingestion, metadata, and query processing are all contained in a single system
- Users only specify data freshness and consistency, and the necessary work is coordinated automatically
- Iceberg, Hudi, and Delta Lake require integrating separate tools for this, which increases complexity
- Intelligent optimization of materialized views and data layout
- Automatically switches between CoW (Copy-on-Write) and MoR (Merge-on-Read) to suit the workload
- Creates and manages snapshots and materialized views appropriately according to user-defined performance and cost requirements
- Hudi, Delta Lake, and Iceberg mostly require this process to be configured manually
- Fine-grained data freshness control
- Like Napa's QT, when users specify the freshness level they want, the system finds the trade-off between performance and cost
- Existing systems relied on periodic snapshots and monitoring, making real-time guarantees difficult
- Advanced indexing and partitioning inspired by Apache Pinot
- Supports advanced index types such as Star-Tree, making even high-cardinality (many-dimension) analytics possible
- Pursues performance gains beyond the simple partitioning and data skipping of Iceberg and Delta Lake
- Flexible configuration
- Multiple consumers can configure the same table to suit their different performance, cost, and freshness needs
- Existing systems have a limited configuration scope, so a lot of manual tuning is needed to handle diverse workloads
- ACID and schema evolution support
- Inherits and extends the ACID and schema evolution foundations of Iceberg and Delta Lake
- Aims to automate concurrent schema changes and data consistency guarantees in distributed environments
- Openness and extensibility
- Supports open standards and integration with various query engines, and can be extended as needed
- Not locked into a specific vendor or platform, allowing features to be applied flexibly according to user needs
Why LakeDB is the next evolution
- Performance: approaches OLAP-level speed through advanced indexes, materialized views, and dynamic data layout optimization
- Simplicity: provides management features in a single system, so you only set user requirements (freshness, performance, etc.) and it adjusts automatically
- Efficiency: reduces resource usage and operational burden, so cost benefits can also be expected
- Flexibility: can handle diverse workloads with fine-grained data freshness control and rich configuration options
- Real-time analytics: adopts Apache Pinot's indexing techniques to pursue both low latency and high throughput
Conclusion
- Apache Iceberg, Apache Hudi, and Delta Lake have played a major role in advancing the Lakehouse concept
- Combining Google's Napa approach with ideas from Apache Pinot and others makes it possible to envision a more unified and powerful LakeDB
- LakeDB, as a mature unified system spanning storage, ingestion, metadata, and query processing, is likely to become the next-generation data management architecture