TY  - JOUR
A1  - Ivanov, Todor
A1  - Pergolesi, Matteo
T1  - The impact of columnar file formats on SQL‐on‐hadoop engine performance: a study on ORC and Parquet
T2  - Concurrency and computation : practice & experience
N2  - Columnar file formats provide an efficient way to store data to be queried by SQL‐on‐Hadoop engines. Related works consider the performance of processing engine and file format together, which makes it impossible to predict their individual impact. In this work, we propose an alternative approach: by executing each file format on the same processing engine, we compare the different file formats as well as their different parameter settings. We apply our strategy to two processing engines, Hive and SparkSQL, and evaluate the performance of two columnar file formats, ORC and Parquet. We use BigBench (TPCx‐BB), a standardized application‐level benchmark for Big Data scenarios. Our experiments confirm that the file format selection and its configuration significantly affect the overall performance. We show that ORC generally performs better on Hive, whereas Parquet achieves best performance with SparkSQL. Using ZLIB compression brings up to 60.2% improvement with ORC, while Parquet achieves up to 7% improvement with Snappy. Exceptions are the queries involving text processing, which do not benefit from using any compression.
KW  - BigBench
KW  - big data benchmarking
KW  - columnar file formats
KW  - Hive
KW  - ORC
KW  - Parquet
KW  - SparkSQL
KW  - SQL-on-Hadoop
Y1  - 2019
UR  - http://publikationen.ub.uni-frankfurt.de/frontdoor/index/index/docId/56316
UR  - https://nbn-resolving.org/urn:nbn:de:hebis:30:3-563167
SN  - 1532-0634
SN  - 1532-0626
VL  - 32.2020
IS  - e5523
PB  - John Wiley & Sons Ltd
CY  - Chichester
ER  -