Hadoop

lastmod 2026-02-22created 2015-03-1222 edits

Apache Hadoop (High-Availability Distributed Object-Oriented Platform) is a freeware Java software framework that supports distributed applications running on large computer clusters that can process massive amounts of data. It was originally developed to support distributed processing for Nutch, and is a subproject of Apache Lucene. It implements the Hadoop Distributed File System (HDFS), which can replace the Google File System, a distributed processing system, and MapReduce.

Categories

• Big data
• HDFS+MapReduce
• Apache Flume
• Oozie
• Sqoop
• HCatalog: table and storage management service for Hadoop data
• Pig: programming and data flow interface for MapReduce
• Hive: a data warehousing solution that creates queries on Hadoop data using HiveQL (HiveQL), an SQL-like language

See also

• Apache
• Spark
• Impala
• Sqoop
• Kafka
• Hive
• Hadoop
• Oozie
• RedPanda
• FastLanes - next-generation big data file format
• Data Lakes

Favorite site

• Wikipedia (en) page on Hadoop
• The era of big data - Hadoop, what is Hadoop..?
• How the Hadoop file system (HDFS) stores files
• KLDP: Apache Hadoop
• What Hadoop is Not
• [Recommended] Youtube: Saenghwal Coding Hadoop - Introduction helloworld
• Getting started with Hadoop
• Hadoop tutorial (2-1) – HDFS
• Introduction to Big Data (3) - MapReduce?
• [Recommended] Hadoop 2.2.0 install on Ubuntu Linux 12.04
• Hadoop: understanding the concepts and installing
• Hadoop: a summary from installation to testing
• Hadoop (Hadoop 2.6.0) follow-along installation
• [Recommended] Hadoop, the key to big data, complete guide 1: concepts and what you need to know ¹
• [Recommended] Hadoop, the key to big data, complete guide 2: Hadoop vs RDBMS cost analysis ²
• [Recommended] Hadoop, the key to big data, complete guide 3: the road from RDBMS to Hadoop ³
• Big data: Hadoop-based large-scale log processing
• [Recommended] (Spark) Understanding Spark
  • (Hadoop) HDFS implementation concepts and design
  • (Hadoop) Understanding MapReduce
  • (Hadoop) Components of the Hadoop distributed processing system
• [Recommended] (Big data) Digging into Hadoop and Apache Spark - brief explanation
• Command-line tools can be 235x faster than a Hadoop cluster (2014) | GeekNews
  • [Original] Command-line Tools can be 235x Faster than your Hadoop Cluster - Adam Drake
  • Processing about 1.75GB of chess game data with command-line tools instead of Hadoop finished in 12 seconds, more than 235x faster than Hadoop's 26 minutes
  • Builds a streaming processing pipeline by combining basic shell commands such as grep, sort, uniq, awk, xargs, and mawk, keeping memory use near zero
  • Raises CPU core utilization and minimizes IO bottlenecks through xargs parallel processing and mawk optimization
  • Cost and maintenance burden are significantly lower than processing the same dataset on a Hadoop cluster (7 c1.medium instances)
  • Shows that efficient data analysis is possible even on a single machine, and warns against the unnecessary use of Big Data tools

References