Ariel Rabkin and Randy H. Katz

EECS Department
University of California, Berkeley
Technical Report No. UCB/EECS-2010-25
March 5, 2010

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-25.pdf

Large Internet services companies like Google, Yahoo, and Facebook use the MapReduce programming model to process log data. MapReduce is designed to work on data stored in a distributed filesystem like Hadoop's HDFS. As a result, a number of companies have developed log collection systems that write to HDFS. These systems have a number of common weaknesses, induced by the semantics of the filesystem. They impose a delay, often several minutes, before data is available for processing. They are difficult to integrate with existing applications. They cannot reliably handle concurrent failures. We present a system, called Chukwa, that adds the needed semantics for log collection and analysis. Chukwa uses an end-to-end delivery model that leverages local on-disk log files when possible, easing integration with legacy systems. Chukwa offers a choice of delivery models, making subsets of the collected data available promptly for clients that require it, while reliably storing a copy in HDFS. We demonstrate that our system works correctly on a 200-node testbed and can collect in excess of 200 MB/sec of log data. We supplement these measurements with a set of case studies.

Advisor: Randy H. Katz

BibTeX citation:

@mastersthesis{Rabkin:EECS-2010-25,
    Author = {Rabkin, Ariel and Katz, Randy H.},
    Title = {Chukwa: A system for reliable large-scale log collection},
    School = {EECS Department, University of California, Berkeley},
    Year = {2010},
    Month = {Mar},
    URL = {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-25.html},
    Number = {UCB/EECS-2010-25},
    Abstract = {Large Internet services companies like Google, Yahoo, and Facebook use the MapReduce programming model to process log data.  MapReduce is designed to work on data stored in a distributed filesystem like Hadoop's HDFS. As a result, a number of companies have developed log collection systems that write to HDFS.  These systems have a number of common weaknesses, induced by the semantics of the filesystem. They impose a delay, often several minutes, before data is available for processing. They are difficult to integrate with existing applications.  They cannot reliably handle concurrent failures. We present a system, called Chukwa, that adds the needed semantics for log collection and analysis. Chukwa uses an end-to-end delivery model that leverages local on-disk log files when possible, easing integration with legacy systems. Chukwa offers a choice of delivery models, making subsets of the collected data available promptly for clients that require it, while reliably storing a copy in HDFS. We demonstrate that our system works correctly on a 200-node testbed and can collect in excess of 200 MB/sec of log data. We supplement these measurements with a set of case studies.}
}

EndNote citation:

%0 Thesis
%A Rabkin, Ariel
%A Katz, Randy H.
%T Chukwa: A system for reliable large-scale log collection
%I EECS Department, University of California, Berkeley
%D 2010
%8 March 5
%@ UCB/EECS-2010-25
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-25.html
%F Rabkin:EECS-2010-25