Antonio Lupher

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2014-1

January 13, 2014

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-1.pdf

Shark is a research data analysis system built on a novel coarse-grained distributed shared-memory abstraction. Shark pairs query processing with deep data analysis, providing a unified system for easy data manipulation using SQL and pushing sophisticated analysis closer to its data. It scales to thousands of nodes in a fault-tolerant manner. Shark can answer queries over 40 times faster than Apache Hive and run machine learning programs on large datasets over 25 times faster than equivalent MapReduce programs on Apache Hadoop. Unlike previous systems, Shark shows that it is possible to achieve these speedups while retaining a MapReduce-like execution engine, with the fine-grained fault tolerance properties that such an engine provides. Shark additionally provides several extensions to its engine, including table and column-level statistics collection as well as a cost-based optimizer, both of which we describe in depth in this paper. Cost-based query optimization in some cases improves the performance of queries with multiple joins by orders of magnitude over Hive and over 2 times compared to previous versions of Shark. The result is a system that matches the reported speedups of MPP analytic databases against MapReduce, while providing more comprehensive fault tolerance and complex analytics capabilities.

Advisors: Scott Shenker

---

BibTeX citation:

@mastersthesis{Lupher:EECS-2014-1,
    Author= {Lupher, Antonio},
    Title= {Shark: SQL and Analytics with Cost-Based Query Optimization on Coarse-Grained Distributed Memory},
    School= {EECS Department, University of California, Berkeley},
    Year= {2014},
    Month= {Jan},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-1.html},
    Number= {UCB/EECS-2014-1},
    Abstract= {Shark is a research data analysis system built on a novel coarse-grained distributed shared-memory abstraction. Shark pairs query processing with deep data analysis, providing a unified system for easy data manipulation using SQL and pushing sophisticated analysis closer to its data. It scales to thousands of nodes in a fault-tolerant manner. Shark can answer queries over 40 times faster than Apache Hive and run machine learning programs on large datasets over 25 times faster than equivalent MapReduce programs on Apache Hadoop. Unlike previous systems, Shark shows that it is possible to achieve these speedups while retaining a MapReduce-like execution engine, with the fine-grained fault tolerance properties that such an engine provides.  Shark additionally provides several extensions to its engine, including table and column-level statistics collection as well as a cost-based optimizer, both of which we describe in depth in this paper.  Cost-based query optimization in some cases improves the performance of queries with multiple joins by orders of magnitude over Hive and over 2 times compared to previous versions of Shark. The result is a system that matches the reported speedups of MPP analytic databases against MapReduce, while providing more comprehensive fault tolerance and complex analytics capabilities.},
}

EndNote citation:

%0 Thesis
%A Lupher, Antonio 
%T Shark: SQL and Analytics with Cost-Based Query Optimization on Coarse-Grained Distributed Memory
%I EECS Department, University of California, Berkeley
%D 2014
%8 January 13
%@ UCB/EECS-2014-1
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-1.html
%F Lupher:EECS-2014-1