Skip to content
Search

Apache Spark

lastmod 2026-01-19created 2021-07-093 edits

Apache Spark is an open-source cluster computing framework. Originally developed at the AMPLab of the University of California, Berkeley, the Spark codebase was later donated to the Apache Software Foundation, which has maintained it ever since. Spark provides an interface for programming entire clusters with implicit data parallelism and fault tolerance.

Spark concepts and key features

These days many people around me are studying Apache Spark, and many are also doing data analysis with Zeppelin, which is built on Spark, so starting today I have begun looking into Spark again.

I have been interested in Spark a few times before. It gets mentioned alongside Storm among data streaming frameworks, its Spark ML library comes up when looking at machine learning frameworks, and when I looked at a certain company's data analysis architecture some time ago, they were analyzing data stored in Cassandra with a framework called Spark/Shark. Some people also call it memory-based Hadoop.

To define Spark in a word, it is a 

범용적 목적의 분산 고성능 클러스터링 플랫폼 (General purpose high performance distributed platform)

That is a long and vague phrase, but there is no better way to describe it.

Spark is a general-purpose distributed platform. It does not only run Map & Reduce like Hadoop, nor does it only do stream processing like Storm. It is simply a general-purpose distributed clustering platform that lets you run computations on multiple distributed nodes, and on top of it you can add modules such as Map & Reduce or stream processing to perform those functions.

In particular, it is also called in-memory Hadoop. Spark started from the goal of speeding up existing Hadoop, which is slow because it performs MR (aka. Map & Reduce) jobs on disk, by moving that work into memory.

Features such as SQL-based querying and streaming were then added on top of that platform, giving Spark its current form.

As mentioned above, Spark's key features are

  • Map & Reduce (cf. Hadoop)
  • Streaming data handling (cf. Apache Storm)
  • SQL-based data queries (cf. Hadoop's Hive)
  • Machine learning library (cf. Apache Mahout)

and so on.

Advantages of Spark

High speed is a given, but when I looked into why everyone is going on about Spark, its nature as a platform turns out to be the advantage. For example, in the implementation of existing big data analysis platforms, batch analysis loaded data with tools such as ETL, refined the data with Hadoop MR, put it into a database such as OLAP, and then visualized it as graphs with a reporting tool. If real-time analysis was needed, Storm was connected to add real-time data analysis. That alone involves several frameworks, and it took a lot of effort to study them and to deploy and operate the system, but with Spark all of this is possible in one.

In other words, once you learn it, you can do many things within a single platform, and it is fast as well. And because a single platform provides various kinds of processing such as batch, streaming, and machine learning, it can process one set of data in many different forms.

It also has advantages in integration. Spark is implemented in Scala, but it has SDKs supporting various languages such as Python, Java, and Scala, and on the data storage side it supports various data stores such as Hadoop, Amazon S3, and Cassandra. Thanks to this integration, you can analyze and process data by connecting various data stores from various languages.

Cluster Mode

Hadoop vs Spark

  1. Hadoop and Apache Spark play different roles. Hadoop and Apache Spark have in common that both are big data frameworks, but their purposes differ considerably. First, Hadoop is essentially a distributed data infrastructure that distributes large data collections across multiple nodes in a cluster of commodity servers. The advantage of this approach is that it reduces the user's cost of buying and maintaining custom-built hardware. Hadoop also has strong market support because it indexes and tracks data, greatly improving the efficiency of big data processing and analytics. Spark, on the other hand, is a data processing tool that runs on top of such distributed data collections, and it does not act as distributed storage.
  2. Hadoop and Apache Spark are independent of each other. In addition to a storage component called the Hadoop Distributed File System (HDFS), Hadoop also provides MapReduce, a processing component. In other words, it does not require Spark for processing. Conversely, Spark can be used without Hadoop. It is true that Spark does not include its own file management system and needs one, but it can be combined with other cloud-based data platforms instead of HDFS. However, since Spark was originally designed as a solution for Hadoop, the two work best together.
  3. Spark is faster. In general, Spark is far faster than MapReduce. The difference comes from how they process data. Unlike MapReduce, which processes data step by step, Spark handles the entire dataset at once. Kirk Borne, principal data scientist at Booz Allen Hamilton, said, "The MapReduce workflow goes like this: read data from the cluster, perform an operation, write the results to the cluster, read the updated data from the cluster again, perform the next operation, and write the results to the cluster. Spark, on the other hand, can complete all data operations in memory in near real time. In other words, the whole process of reading data from the cluster, performing all the required analytic operations, and writing the results to the cluster happens concurrently." Borne explained that for batch processing Spark can run 10 times faster than MapReduce, and for in-memory analytics the speed difference reaches 100 times.
  4. You don't necessarily need Spark's speed. If most of your data operations and reporting requirements are static and you can wait for batch-mode processing, MapReduce's processing approach will not be a problem. Spark is really needed when processing streaming data generated by things like sensors in a factory, or when an application requires complex operations such as machine learning algorithms. Specifically, Spark helps a lot with applications such as real-time marketing campaigns, online product recommendations, cybersecurity analytics, and machine log monitoring.
  5. Their failure recovery approaches differ, but both are decent. Hadoop's approach of writing results to disk after every operation can be very useful in case of system crashes or failures. Spark, for its part, ensures resilience by distributing data objects across the data cluster in the form of Resilient Distributed Datasets (RDD). Borne explained, "These data objects can be stored in memory or on disk, and RDD is a technology that allows full recovery even in case of crashes or failures."

See also

Favorite site

Image Processing

author Zer0 sitemap