Skip to content
Search

Big data

lastmod 2026-02-22created 2014-12-0315 edits

Big data refers to large sets of structured or unstructured data that exceed the capacity of conventional database management tools to collect, store, manage, and analyze, as well as the technology for extracting value from such data and analyzing the results.

The development of big data technology, characterized by the generation, collection, analysis, and representation of large-scale data of many kinds, makes it possible to predict a diversified modern society more accurately and make it run efficiently, to provide, manage, and analyze tailored information for each member of an individualized modern society, and sometimes to realize technologies that were impossible in the past.

In this way, big data offers the possibility of providing valuable information to society and humanity across all areas, including politics, society, the economy, culture, and science and technology, and its importance is being highlighted.

However, the problems of big data lie in privacy invasion and security. Big data is a collection of vast amounts of information about vast numbers of individuals. So when big data is collected and analyzed, it can turn into a Big Brother that collects and manages even individuals' private information. And if the data gathered this way is leaked because of a security problem, this too can be a big problem, since it means the information of almost everyone is leaked.

The World Economic Forum selected big data technology as the first of its top 10 emerging technologies in 2012, and the Office of Strategic R&D Planning of Korea's Ministry of Knowledge Economy selected big data as one of the top 10 core IT technologies; the world has recently been paying close attention to big data.

Categories

Bigdata platform

Big data platforms, typified by the Hadoop and Spark ecosystems, emerged to process large volumes of data in many different forms. They can collect structured, semi-structured, and unstructured data alike.

A big data platform is a data processing environment that enables new insights and business value by supporting roles such as data collection, processing, storage and management, analysis, and visualization.

Bigdata platform vs Data warehouse

  • Data coverage — Structured, semi-structured (JSON, Parquet), and unstructured (text, images, logs) data can all be stored in their raw form. With the Schema-on-Read approach, the schema is applied when the data is read.
  • Scalability — Because it uses a distributed architecture that scales out commodity hardware horizontally (scale-out), even petabytes of data or more can be stored and processed relatively cheaply. Traditional DWs, by contrast, rely mainly on vertical scaling (scale-up), so costs rise sharply.
  • Processing — In addition to batch processing, it also supports streaming (Kafka, Flink, Spark Streaming) processing, enabling near-real-time analysis. DWs mainly focus on batch-based OLAP analysis.
  • Cost structure — Much of it is open-source based, so license costs are low, but operational complexity and staffing costs are high. DWs have high license/service costs, but operation is relatively simple.

Modern convergence: Korean: 데이터 레이크하우스

Recently, the boundary between the two has been breaking down.

Data lakehouses (Databricks Delta Lake, Apache Iceberg, Apache Hudi) are an architecture that combines the flexible storage capabilities of a data lake with the ACID transactions and schema management of a DW.

Cloud DWs such as Snowflake and BigQuery are also absorbing the advantages of big data platforms by adopting semi-structured data support and a decoupled storage-compute architecture.

Understanding real-time data concepts

Before learning about streaming data processing, let's get familiar with a few terms and concepts.

  • Bounded data processing : Processing data that is already stored; no special processing pattern
  • Unbounded data : Streaming data, processed with two patterns: batch and streaming.
    • Batch : Collects streaming data in fixed time units and processes it. Simple to implement, but since processing happens after the data is collected, it falls short of real-time processing.
    • Streaming : Streaming processing includes approaches such as Time agnostic, Filtering, Inner join, and Windowing, and is more complex than batch. This is because skew varies greatly with the environment, so the order in which data arrives at the system is not sequential either.
      • Time agnostic : Data without a time attribute, processed in the order it arrives.
      • Filtering : Filters only specific data from the incoming data, then stores it.
      • Inner join : As the word intersection suggests, compares and matches the values coming in from two unbounded data sources to tell values apart.
      • Windowing : Processes streaming data at fixed time intervals.

Big data is dead

Keep in mind when reading that the company that wrote this is MotherDuck, the company that makes DuckDB.

  • Who am I and why do I care → An early engineer on BigQuery
  • The obligatory intro slide → The "data explodes over time" graph everyone cites
  • Most people don't have that much data
  • Storage and compute are separated, with a bias toward storage
  • Workload sizes are smaller than total data size
  • Most data is rarely queried
  • The big data frontier keeps receding
  • Data is a liability (Liability)
    • Another definition of big data is "when the cost of keeping data is less than the cost of figuring out what to throw away"
  • Are you in the big data one percent?
    • Are you really generating huge amounts of data?
    • If so, do you really need to use huge amounts of data at once?
    • If so, is the data really too big to fit on one system?
    • If so, are you sure you're not just a data hoarder (Hoarder)?
    • If so, wouldn't it be better to summarize?
  • If you answer no to even one of the questions above,
  • then rather than "big data at a scary scale" that you may or may not ever have,
  • it may be better to use "next-generation data tools that let you handle the data scale you actually have"

Solutions

The process of collecting and processing data and using it to generate revenue involves complex and cumbersome steps, and new technologies for real-time data analysis have emerged to handle this faster. A brief summary follows.

Queuing system

Stream processing

  • Storm
  • Spark Streaming
  • Samza

High-speed storage

See also

Favorite site

author Zer0 sitemap