Skip to content
Search

Data Lakehouse

lastmod 2026-02-22created 2025-02-0214 edits

Data Lakehouse

In a word, it is an architecture that combines only the strengths of a Korean: 데이터 레이크 and a Korean: 데이터 웨어하우스.

Categories

What is a data lakehouse?

A data lakehouse adds to the flexibility, cost efficiency, and large-scale support of a data lake,

integrating the data management features and ACID transactions of a data warehouse, as a new kind of open data management architecture

that supports business intelligence (BI) and machine learning (ML) across all data.

Why did it emerge?

Previously, this structure was typical:

This means the data is managed in two copies, the pipelines get complicated, and the cost doubles. The lakehouse is an attempt to unify this 2-tier structure into 1 tier.

Core idea

It puts a table format layer on top of the data lake's cheap object storage (S3, GCS, etc.) to provide DW-level features.

  • Taken from the lake — low-cost storage, accepts all data types (structured/semi-structured/unstructured), open formats (Parquet, ORC)
  • Taken from the warehouse — ACID transactions, schema management and evolution, time travel (querying data as of a past point in time), SQL-based analytics, data governance

Data lakehouse: simplicity, flexibility, and low cost

A data lakehouse is a new, open system design. It implements data structures and data management features similar to a data warehouse, but directly on the low-cost storage used for data lakes. Merging the two into one speeds up data teams' work, because they can use data without accessing multiple systems. A data lakehouse also lets team members carry out data science, machine learning, and business analytics projects using the most complete and up-to-date version of the data.

Data-lakehouse-new.png

Data lakehouse vs. data lake vs. data warehouse

'Data lakehouse' is a term that combines two existing types of data storage: the data warehouse and the data lake. So what are the differences between a data lakehouse, a data lake, and a data warehouse?

Data warehouse
A data warehouse gives business users who need to produce reports and statistics for decision making fast data access and SQL compatibility. All data must go through an ETL (extract, transform, load) step. That is, it is optimized for a specific format or schema based on the use case before loading, supporting high-performance queries and data integrity. However, this approach limits the flexibility of data access and incurs extra cost when data has to be migrated for future use.
Data lake
A data lake stores large amounts of structured and unstructured data in its native format. Unlike a data warehouse, data is processed, cleaned, and transformed during analysis, which makes loading faster, so it is ideal for big data processing, machine learning, or predictive analytics. However, because the group able to use the data is limited to users with data science expertise, data quality can degrade over time if it is not properly maintained by such people. Also, with a data lake, the data is not processed, which makes real-time queries harder, so cleaning, processing, ingestion, and integration may be needed before the data can be used.
Data lakehouse
A data lakehouse merges these two approaches to create a single structure in which data can be accessed and used for a variety of purposes, from BI to data science, AI, and machine learning. That is, a data lakehouse captures all of an organization's structured, unstructured, and semi-structured data and stores it in inexpensive storage, while giving all users the ability to organize and explore the data as needed.

Key enabling technologies of the data lakehouse

Several technical advances have made the data lakehouse possible.

  • A metadata layer for data lakes
  • New query engine designs that enable high-performance SQL execution on data lakes
  • Optimized access for data science and machine learning tools, and so on.

Roadmap: Data 3.0 in the Lakehouse Era

Enterprise data infrastructure evolves along with technological progress, enabling new products and services. Data infrastructure has developed from traditional on-premises data warehouses to cloud-based data warehouses and data lakes. Recently, with the rapid advance of AI, a new architecture called the data lakehouse has risen, ushering in the Data 3.0 era. The lakehouse is a high-performance, interoperable unified platform supporting diverse uses such as analytics and AI workloads, reimagining the core of enterprise data infrastructure.

Background of the lakehouse innovation

  • From 2019 to 2024, enterprise investment in data infrastructure doubled from about $180 billion to $350 billion
  • Existing data warehouses and data lakes do not fully meet the requirements of AI
  • AI-centric workloads have the following requirements:
    • Must handle structured, semi-structured, and unstructured data alike
    • Must support real-time, multimodal, composable data processing
    • Need interoperability between traditional databases and vector databases
  • Changing demands of enterprise customers:
    • Demand for eliminating data duplication
    • Growing complexity of data governance
    • Demand for escaping vendor lock-in and for flexibility
    • Difficulty finding solutions suited to AI

Open table formats make the lakehouse possible

  • Open table formats (OTF) such as Delta Lake, Iceberg, and Hudi form the foundation of the lakehouse
  • Key features:
    • ACID transaction support: guarantees data consistency and reliability
    • Supports batch and streaming processing
    • Provides schema and partition flexibility
    • Time travel lets you restore to a previous state
    • Scalable metadata management

The rise of the lakehouse paradigm

  • The data lakehouse is a new architecture that combines the performance of a data warehouse with the flexibility of a data lake
  • Emerging as next-generation infrastructure for AI-based applications, real-time analytics, and enterprise intelligence
  • Major companies and startups are rushing to move to the lakehouse, and a new related market is forming

Thesis 1: AI-driven ingestion and transformation enable intelligent real-time pipelines

  • Existing ETL tools are inefficient at AI scale
  • Prefect, Windmill, dltHub, and others support code-based data pipelines and orchestration
  • Tools like Tobiko provide SQL automation, data lineage, dependency tracking, and more
  • Anthropic's Model Context Protocol (MCP) provides a standardized interface for maintaining context in AI workflows
  • Apache Kafka and Flink provide messaging and stream processing capabilities essential for real-time model training and inference
  • Chalk AI contributes to fast decision making by providing a real-time inference platform
  • The metadata layer is emerging as an important source of truth in the AI era

Thesis 2: The strategic importance of the metadata layer comes to the fore

  • Metadata is now not just information but a central layer that drives action
  • Open table formats such as Iceberg, Delta Lake, and Hudi lead metadata innovation
  • Lakehouse-native catalogs such as Datastrato and Vakamo are appearing
  • Acryl Data's DataHub supports data access and governance for both humans and AI agents
  • OpenHouse, Apache Amoro, Ryft, and others provide a metadata-centric control plane
  • Flarion.io, Greybeam, and others are developing performance optimization tools in layers other than storage

Thesis 3: Changes in compute and query engines

  • With the spread of the lakehouse, the shift is from a single-platform-centric structure to a modular architecture
  • Besides Snowflake and Databricks, specialized solutions such as DuckDB, ClickHouse, and Druid are growing
  • Daft, typedef, Mooncake, Bauplan, and others are developing new compute frameworks for AI-centric optimization
  • The emergence of AI-optimized query engines and federated compute platforms is setting a new standard for data processing

Thesis 4: The line between data engineering and software engineering is blurring

  • AI-centric applications are shifting all developers toward data-centric skills
  • dbt Labs brings software engineering practices such as version control, testing, and CI/CD to data development
  • Gable supports building data pipelines with a user-friendly interface
  • Temporal and Inngest secure reliability and observability for complex distributed workflows
  • Contributions to open source are surging, and data-related projects on GitHub are growing faster than general software
  • Open-source adoption is increasing to get better support from LLMs
  • As AI and data-centric engineering converge, team structures and development practices are changing fundamentally

Practical caveats

It is not a silver bullet.

For pure SQL analytics performance, dedicated DWs like Snowflake or BigQuery are still often faster, and the operational complexity is considerable.

So in practice, hybrid setups are also common: the lakehouse sits at the center, with a dedicated DW alongside for specific workloads.

See also

Favorite site

author Zer0 sitemap