Data Lakes
lastmod 2026-07-20created 2025-02-029 edits
A data lake is a system or repository of data stored in its natural/raw format, usually object blobs or files. A data lake is usually a single store of data including raw copies of source system data, sensor data, social data, etc., and transformed data used for tasks such as reporting, visualization, advanced analytics, and machine learning. A data lake can include structured data from relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured data (emails, documents, PDFs), and binary data (images, audio, video). A data lake can be established "on premises" (within an organization's data centers) or "in the cloud" (using cloud services from vendors such as Amazon, Microsoft, Oracle Cloud, or Google).
About
A large-scale store that keeps all kinds of data in its original form. As the name suggests, the idea is that data flows in and accumulates like a "lake".
Background
James Dixon, then chief technology officer at Pentaho, coined the term in 2011 to contrast it with a data mart, which is a smaller repository of interesting attributes derived from raw data. In promoting data lakes, he argued that data marts have several inherent problems, such as information siloing. PricewaterhouseCoopers (PwC) said that data lakes could "put an end to data silos". In their study on data lakes, they noted that enterprises were "starting to extract and place data for analytics into a single, Korean: 하둡-based repository".
Key features
- Stored as-is — CSV, JSON, Parquet, images, video, log files, and so on are simply dumped in without transformation. There is no need to define a schema or structure in advance (Schema-on-Read).
- Low-cost, high-capacity storage — Because it uses object/distributed storage such as S3, GCS, Azure Blob, and HDFS, even petabyte-scale data can be kept relatively cheaply.
- Diverse consumption patterns — The same data can be used for many purposes, such as SQL analytics, ML training, and stream processing.
The data lake pitfall: Data Swamp
The biggest problem is that without management it turns into a data swamp (Data Swamp).
If you pour in any data indiscriminately, you end up in a state where nobody knows what is where, what its quality is, or who put it in and when.
That is why metadata management, a data catalog, and governance are essential, and the Korean: 레이크하우스 architecture is an attempt to solve this problem structurally.
Categories
- Lakehouse
- DuckLake - Unified data lake and catalog format
- LakeDB
- Rill - BI tool optimized to shorten the time from data lake to dashboard