Yanpei Chen and Archana Sulochana Ganapathi and Rean Griffith and Randy H. Katz

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2010-95

June 14, 2010

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-95.pdf

System designers in industry are often overwhelmed by large scale data, while researchers in academic often confront a lack of publicly available production data. In this paper, we analyze a large scale production workload trace recently made publicly available by Google. We offer a statistical profile of the data, with several interesting discoveries regarding job arrival patterns, CPU and memory consumptions, task durations, and others. We further perform k-means clustering to identify common groups of jobs, with several methodological departures and different findings compared with prior work on similar data. We also do correlation analysis between job semantics and job behavior, leading to helpful perspectives on capacity planning and system tuning. Our key finding is that while the limited dataset size prevents us from generalizing the trace behaviors observed, the analytical methods we describe nonetheless allow us to extract many system design insights.

---

BibTeX citation:

@techreport{Chen:EECS-2010-95,
    Author= {Chen, Yanpei and Ganapathi, Archana Sulochana and Griffith, Rean and Katz, Randy H.},
    Title= {Analysis and Lessons from a Publicly Available Google Cluster Trace},
    Year= {2010},
    Month= {Jun},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-95.html},
    Number= {UCB/EECS-2010-95},
    Abstract= {System designers in industry are often overwhelmed by large scale data, while researchers in academic often confront a lack of publicly available production data. In this paper, we analyze a large scale production workload trace recently made publicly available by Google. We offer a statistical profile of the data, with several interesting discoveries regarding job arrival patterns, CPU and memory consumptions, task durations, and others. We further perform k-means clustering to identify common groups of jobs, with several methodological departures and different findings compared with prior work on similar data. We also do correlation analysis between job semantics and job behavior, leading to helpful perspectives on capacity planning and system tuning. Our key finding is that while the limited dataset size prevents us from generalizing the trace behaviors observed, the analytical methods we describe nonetheless allow us to extract many system design insights.},
}

EndNote citation:

%0 Report
%A Chen, Yanpei 
%A Ganapathi, Archana Sulochana 
%A Griffith, Rean 
%A Katz, Randy H. 
%T Analysis and Lessons from a Publicly Available Google Cluster Trace
%I EECS Department, University of California, Berkeley
%D 2010
%8 June 14
%@ UCB/EECS-2010-95
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-95.html
%F Chen:EECS-2010-95