Decentralized Systems Need Decentralized Benchmarks
David Oppenheimer and David A. Patterson and Joseph M. Hellerstein
EECS Department, University of California, Berkeley
Technical Report No. UCB/CSD-03-1234
, 2003
http://www2.eecs.berkeley.edu/Pubs/TechRpts/2003/CSD-03-1234.pdf
Decentralized systems require new benchmarks and new benchmarking techniques. We propose a general methodology for benchmarking the performability of one class of decentralized system: peer-to-peer applications built on top of distributed hash tables (DHTs). Furthermore, we argue that benchmarks for decentralized systems must be designed and implemented with similar concern for scalability and robustness as the systems they are designed to benchmark, implying a need for decentralized load generation, fault injection, and metric collection. These criteria lead us to propose a benchmark implementation that uses a DHT to publish the faultload description and to store collected metrics, and uses a DHT-based relational query engine to analyze benchmark results. Finally, we argue that the fault injection and monitoring mechanisms required to run such benchmarks are reusable for online robustness testing, problem detection, and problem diagnosis, and that they therefore should be provided as infrastructure services.
BibTeX citation:
@techreport{Oppenheimer:CSD-03-1234, Author= {Oppenheimer, David and Patterson, David A. and Hellerstein, Joseph M.}, Title= {Decentralized Systems Need Decentralized Benchmarks}, Year= {2003}, Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2003/5221.html}, Number= {UCB/CSD-03-1234}, Abstract= {Decentralized systems require new benchmarks and new benchmarking techniques. We propose a general methodology for benchmarking the performability of one class of decentralized system: peer-to-peer applications built on top of distributed hash tables (DHTs). Furthermore, we argue that benchmarks for decentralized systems must be designed and implemented with similar concern for scalability and robustness as the systems they are designed to benchmark, implying a need for decentralized load generation, fault injection, and metric collection. These criteria lead us to propose a benchmark implementation that uses a DHT to publish the faultload description and to store collected metrics, and uses a DHT-based relational query engine to analyze benchmark results. Finally, we argue that the fault injection and monitoring mechanisms required to run such benchmarks are reusable for online robustness testing, problem detection, and problem diagnosis, and that they therefore should be provided as infrastructure services.}, }
EndNote citation:
%0 Report %A Oppenheimer, David %A Patterson, David A. %A Hellerstein, Joseph M. %T Decentralized Systems Need Decentralized Benchmarks %I EECS Department, University of California, Berkeley %D 2003 %@ UCB/CSD-03-1234 %U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2003/5221.html %F Oppenheimer:CSD-03-1234