Econometrics at scale: Spark up big data in economics

  • This paper provides an overview of how to use "big data" for economic research. We investigate the performance and ease of use of different Spark applications running on a distributed file system to enable the handling and analysis of data sets which were previously not usable due to their size. More specifically, we explain how to use Spark to (i) explore big data sets which exceed retail grade computers memory size and (ii) run typical econometric tasks including microeconometric, panel data and time series regression models which are prohibitively expensive to evaluate on stand-alone machines. By bridging the gap between the abstract concept of Spark and ready-to-use examples which can easily be altered to suite the researchers need, we provide economists and social scientists more generally with the theory and practice to handle the ever growing datasets available. The ease of reproducing the examples in this paper makes this guide a useful reference for researchers with a limited background in data handling and distributed computing.

Download full text files

Export metadata

Metadaten
Author:Benjamin Bluhm, Jannic Cutura
URN:urn:nbn:de:hebis:30:3-518045
DOI:https://doi.org/10.2139/ssrn.3226976
Parent Title (English):SAFE working paper series ; No. 266
Series (Serial Number):SAFE working paper (266)
Publisher:SAFE
Place of publication:Frankfurt am Main
Document Type:Working Paper
Language:English
Year of Completion:2020
Year of first Publication:2020
Publishing Institution:Universit├Ątsbibliothek Johann Christian Senckenberg
Release Date:2020/02/10
Tag:Apache Spark; Distributed Computing; Econometrics
Issue:February 6, 2020
Page Number:49
Note:
An earlier version of this paper circulated as "Time Series Econometrics at Scale: A Practical Guide to Parallel Computing in (Py)Spark".
HeBIS-PPN:460762621
Institutes:Wirtschaftswissenschaften / Wirtschaftswissenschaften
Wissenschaftliche Zentren und koordinierte Programme / House of Finance (HoF)
Wissenschaftliche Zentren und koordinierte Programme / Center for Financial Studies (CFS)
Wissenschaftliche Zentren und koordinierte Programme / Sustainable Architecture for Finance in Europe (SAFE)
Dewey Decimal Classification:3 Sozialwissenschaften / 33 Wirtschaft / 330 Wirtschaft
JEL-Classification:C Mathematical and Quantitative Methods / C5 Econometric Modeling / C53 Forecasting and Other Model Applications
Sammlungen:Universit├Ątspublikationen
Licence (German):License LogoDeutsches Urheberrecht