PreFail: A Programmable Failure-Injection Framework

Pallavi Joshi, Haryadi S. Gunawi and Koushik Sen

EECS Department
University of California, Berkeley
Technical Report No. UCB/EECS-2011-30
April 22, 2011

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2011/EECS-2011-30.pdf

As hardware failures are no longer rare in the era of cloud computing, cloud software systems must ``prevail'' against multiple, diverse failures that are likely to occur. Testing software against multiple failures poses the problem of combinatorial explosion of multiple failures. To address this, a tester can write diverse policies that prune down the space of multiple failures while meeting her testing objective. In this paper, we present PreFail, a programmable failure-injection framework that enables testers to write a wide range of pruning policies. Using the principle of separation of mechanism and policy, we decouple a failure-injection framework into two components: failure-injection engine and driver. The policies written in the driver decide which failures should be injected by the engine. We define clear abstractions on which the two components interact. We integrate PreFail to three cloud software systems, show a wide variety of pruning policies that we can write for them and the speed-ups that we obtain with those policies.


BibTeX citation:

@techreport{Joshi:EECS-2011-30,
    Author = {Joshi, Pallavi and Gunawi, Haryadi S. and Sen, Koushik},
    Title = {PreFail: A Programmable Failure-Injection Framework},
    Institution = {EECS Department, University of California, Berkeley},
    Year = {2011},
    Month = {Apr},
    URL = {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2011/EECS-2011-30.html},
    Number = {UCB/EECS-2011-30},
    Abstract = {As hardware failures are no longer rare in the era
of cloud computing, cloud software systems must
``prevail'' against multiple, diverse failures
that are likely to occur.  Testing software
against multiple failures poses the problem of
combinatorial explosion of multiple failures.  To
address this, a tester can write diverse policies
that prune down the space of multiple failures
while meeting her testing objective.  In this
paper, we present PreFail, a programmable
failure-injection framework that enables testers
to write a wide range of pruning policies.  Using
the principle of separation of mechanism and
policy, we decouple a failure-injection framework
into two components: failure-injection engine and
driver. The policies written in the driver decide
which failures should be injected by the engine.
We define clear abstractions on which the two
components interact.  We integrate PreFail to
three cloud software systems, show a wide variety
of pruning policies that we can write for them and
the speed-ups that we obtain with those policies.}
}

EndNote citation:

%0 Report
%A Joshi, Pallavi
%A Gunawi, Haryadi S.
%A Sen, Koushik
%T PreFail: A Programmable Failure-Injection Framework
%I EECS Department, University of California, Berkeley
%D 2011
%8 April 22
%@ UCB/EECS-2011-30
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2011/EECS-2011-30.html
%F Joshi:EECS-2011-30