Zachary Golan-Strieb and David A. Wagner

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2021-241

December 1, 2021

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-241.pdf

Current machine learning models suffer from evasion at- tacks such as adversarial examples. This introduces security and safety concerns that lack any clear solution. Recently, the use of random transformations (RT) has emerged as a promising defense against adversarial examples. However, it has not been rigorously evaluated, and its effectiveness is not well-understood. In this paper, we attempt to construct the strongest possible RT defense through the informed selection of transformations and the use of Bayesian optimization to tune their parameters. Furthermore, we attempt to identify the strongest possible attack to evaluate our RT defense. Our new attack vastly outperforms the naive attack, reducing the accuracy of our model by an additional 30%. In the process of formulating our defense and attack, we perform several ablation studies for both problems, drawing insights that we hope will broadly benefit scientific communities that study stochastic neural networks and robustness properties.

Advisors: David A. Wagner

---

BibTeX citation:

@mastersthesis{Golan-Strieb:EECS-2021-241,
    Author= {Golan-Strieb, Zachary and Wagner, David A.},
    Title= {Towards Evaluating and Understanding the Adversarial Robustness of Random Transformation Defenses},
    School= {EECS Department, University of California, Berkeley},
    Year= {2021},
    Month= {Dec},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-241.html},
    Number= {UCB/EECS-2021-241},
    Abstract= {Current machine learning models suffer from evasion at- tacks such as adversarial examples. This introduces security and safety concerns that lack any clear solution. Recently, the use of random transformations (RT) has emerged as a promising defense against adversarial examples. However, it has not been rigorously evaluated, and its effectiveness is not well-understood. In this paper, we attempt to construct the strongest possible RT defense through the informed selection of transformations and the use of Bayesian optimization to tune their parameters. Furthermore, we attempt to identify the strongest possible attack to evaluate our RT defense. Our new attack vastly outperforms the naive attack, reducing the accuracy of our model by an additional 30%. In the process of formulating our defense and attack, we perform several ablation studies for both problems, drawing insights that we hope will broadly benefit scientific communities that study stochastic neural networks and robustness properties.},
}

EndNote citation:

%0 Thesis
%A Golan-Strieb, Zachary 
%A Wagner, David A. 
%T Towards Evaluating and Understanding the Adversarial Robustness of Random Transformation Defenses
%I EECS Department, University of California, Berkeley
%D 2021
%8 December 1
%@ UCB/EECS-2021-241
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-241.html
%F Golan-Strieb:EECS-2021-241