Presto
lastmod 2018-08-17created 2017-01-221 edits
Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes.
Presto was designed and written from the ground up for interactive analytics and approaches the speed of commercial data warehouses while scaling to the size of organizations like Facebook.
What is Presto
- ZDNET - Facebook releases query engine for 250-petabyte-class DW
- Facebook releases open-source DW engine ‘Presto’… for analyzing 300PB of data
Facebook has released a data warehouse (DW) engine that can query 250-petabyte (PB)-scale data with the SQL language. It is called 'Presto'. It is an SQL-on-Hadoop technology that competes with Cloudera Impala. According to foreign media on the 7th (local time), Facebook unveiled the query engine 'Presto', which can run analytical queries with SQL on a 250PB-scale DW, at a developer conference held at its headquarters that day. Presto is currently used daily by 850 employees within Facebook. The employees scan 350 terabytes (TB) a day. longdesc=Generally, the way to query data in Hadoop with SQL is to use Hive. Hive has the advantage that existing analysts can use it easily because it supports ANSI-like SQL queries, but it lags behind massively parallel processing (MPP)-based DW systems in query processing time. This is because Hive adds an intermediate step of converting SQL queries into MapReduce jobs. Technologies that overcome Hive's limits and enable fast SQL queries form a trend called 'SQL-on-Hadoop'. They are being developed extensively in the Hadoop camp. Cloudera's Impala, released early last month, is a representative engine. 'Tajo', under development by a research team at Korea University and the Hadoop specialist company Gruter in Korea, is also one of them. Presto is likewise a separate SQL query engine that does not use Hive. Without using MapReduce, it directly queries data stored in the Hadoop Distributed File System (HDFS) with SQL statements. Facebook engineer Martin Traverso said, "Historically, our data scientists and analysts have relied on Hive for data analysis," adding, "The problem with Hive is that it was designed for batch processing." He added, "We have several tools that are faster than Hive, but they are also limited in functionality (functionality) and too simple to manage a huge DW," and "Over the past few months, we have been developing Presto to fundamentally close this gap." Facebook created Hive years ago and gave it to the Hadoop camp. But over time, the performance limits of Hive, which depends on MapReduce, became apparent. Performance that takes hours to scan an entire dataset is not ideal when you want near-real-time queries and answers. Presto returns simple queries in a few hundred milliseconds, while complex queries are processed within minutes. Traverso explained, "It is processed in memory and never writes to disk." Presto is not a commercial product. Conceptually it is the same as Impala, but it is optimized for fast performance at the scale of Facebook's actual data nodes. Traverso said, "We plan to release Presto as open source sometime this fall." Facebook engineering manager Ravi Murthy emphasized, "Our DW scale has grown faster than the number of users, becoming 4,000 times larger than four years ago," and "With this growth, we predict it is clear that the DW will reach exabyte scale in a few years." He stressed, "Looking at this exabyte scale, we have to rethink many other things." Traverso explained that besides processing queries fast, Presto is "7 times more efficient than Hive in terms of CPU."