Search engine for discovering works of Art, research articles, and books related to Art and Culture
Javascript must be enabled to continue!
SQL-on-Hadoop
View through CrossRef
SQL query processing for analytics over Hadoop data has recently gained significant traction. Among many systems providing some SQL support over Hadoop, Hive is the first native Hadoop system that uses an underlying framework such as MapReduce or Tez to process SQL-like statements. Impala, on the other hand, represents the new emerging class of SQL-on-Hadoop systems that exploit a shared-nothing parallel database architecture over Hadoop. Both systems optimize their data ingestion via columnar storage, and promote different file formats: ORC and Parquet. In this paper, we compare the performance of these two systems by conducting a set of cluster experiments using a TPC-H like benchmark and two TPC-DS inspired workloads. We also closely study the I/O efficiency of their columnar formats using a set of micro-benchmarks. Our results show that Impala is 3.3
X
to 4.4
X
faster than Hive on MapReduce and 2.1
X
to 2.8
X
than Hive on Tez for the overall TPC-H experiments. Impala is also 8.2
X
to 10
X
faster than Hive on MapReduce and about 4.3
X
faster than Hive on Tez for the TPC-DS inspired experiments. Through detailed analysis of experimental results, we identify the reasons for this performance gap and examine the strengths and limitations of each system.
Association for Computing Machinery (ACM)
Title: SQL-on-Hadoop
Description:
SQL query processing for analytics over Hadoop data has recently gained significant traction.
Among many systems providing some SQL support over Hadoop, Hive is the first native Hadoop system that uses an underlying framework such as MapReduce or Tez to process SQL-like statements.
Impala, on the other hand, represents the new emerging class of SQL-on-Hadoop systems that exploit a shared-nothing parallel database architecture over Hadoop.
Both systems optimize their data ingestion via columnar storage, and promote different file formats: ORC and Parquet.
In this paper, we compare the performance of these two systems by conducting a set of cluster experiments using a TPC-H like benchmark and two TPC-DS inspired workloads.
We also closely study the I/O efficiency of their columnar formats using a set of micro-benchmarks.
Our results show that Impala is 3.
3
X
to 4.
4
X
faster than Hive on MapReduce and 2.
1
X
to 2.
8
X
than Hive on Tez for the overall TPC-H experiments.
Impala is also 8.
2
X
to 10
X
faster than Hive on MapReduce and about 4.
3
X
faster than Hive on Tez for the TPC-DS inspired experiments.
Through detailed analysis of experimental results, we identify the reasons for this performance gap and examine the strengths and limitations of each system.
Related Results
Optimizing Text-to-SQL Transformations: The Potential of Skeleton Decoupling in SKT-SQL
Optimizing Text-to-SQL Transformations: The Potential of Skeleton Decoupling in SKT-SQL
Abstract
The Text-to-SQL technology faces significant challenges in converting natural language questions into SQL code, particularly in handling complexities and diversiti...
Integration of SQL Injection Prevention Methods
Integration of SQL Injection Prevention Methods
In everybody’s life including the organisations, database plays a very important role, since today everything is connected via the Internet. There is a need for a database that h...
SQL INJECTION ATTACKS DETECTION: A PERFORMANCE COMPARISON ON MULTIPLE CLASSIFICATION MODELS
SQL INJECTION ATTACKS DETECTION: A PERFORMANCE COMPARISON ON MULTIPLE CLASSIFICATION MODELS
SQL injection attacks are a common and serious security threat to web applications, where malicious users exploit vulnerabilities to gain unauthorized access to sensitive data or m...
Atomic quantum metrology with narrowband entangled and squeezed states of light
Atomic quantum metrology with narrowband entangled and squeezed states of light
The use of light, especially of laser light, is in many cases the most sensitive way to perform measurements. However, the highest sensitivity that can be achieved with laser light...
Hadoop Tools
Hadoop Tools
As the name indicates, this chapter explains the various additional tools provided by Hadoop. The additional tools provided by Hadoop distribution are Hadoop Streaming, Hadoop Arch...
Enhancing Big Data Security in Hadoop using Machine Learning
Enhancing Big Data Security in Hadoop using Machine Learning
In the era of Big Data, where vast amounts of information are generated and analysed to extract valuable insights, ensuring the security of data has become paramount. Hadoop, as a ...
Hadoop Distributed File System for Big data analysis
Hadoop Distributed File System for Big data analysis
Abstract
Hadoop is framework that is processing data with large volume that cannot be processed by conventional systems. Hadoop has management file system called Hadoop Dis...
Analisis Data Sosial Media Twitter Menggunakan Hadoop dan Spark
Analisis Data Sosial Media Twitter Menggunakan Hadoop dan Spark
Big data merupakan sumber data yang memiliki volume yang besar, variasi yang banyak, dan aliran data yang sangat cepat. Contoh big data antara lain data dari media sosial d...