Kesav Viswanadha

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2021-103

May 14, 2021

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-103.pdf

As autonomous vehicle (AV) technology grows more widespread, questions still persist about how to effectively verify their safety. Much progress has been made in developing testing methodologies such as falsification for autonomous vehicles that interface with driving simulators to generate rich sets of scenarios. We present extensions to the Scenic scenario specification language and VerifAI toolkit that improve the scalability of such methods by allowing for falsification to be done more efficiently and with more complex models of the end goal. We first present a parallelized framework that is interfaced with both the simulation and sampling capabilities of Scenic and the falsification capabilities of VerifAI, reducing the execution time bottleneck inherently present in simulation-based testing. We then present an extension of VerifAI's falsification algorithms to support multi-dimensional objective optimization during sampling, using the concept of rulebooks to specify multiple metrics and a preference ordering over the metrics that can be used to guide the counterexample search process. Lastly, we evaluate the benefits of these extensions with a comprehensive set of experiments written in the Scenic language.

Advisors: Sanjit A. Seshia

---

BibTeX citation:

@mastersthesis{Viswanadha:EECS-2021-103,
    Author= {Viswanadha, Kesav},
    Title= {Scalable Techniques for Sampling-Based Falsification of AI-Based Cyber Physical Systems},
    School= {EECS Department, University of California, Berkeley},
    Year= {2021},
    Month= {May},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-103.html},
    Number= {UCB/EECS-2021-103},
    Abstract= {As autonomous vehicle (AV) technology grows more widespread, questions still persist about how to effectively verify their safety. Much progress has been made in developing testing methodologies such as falsification for autonomous vehicles that interface with driving simulators to generate rich sets of scenarios. We present extensions to the Scenic scenario specification language and VerifAI toolkit that improve the scalability of such methods by allowing for falsification to be done more efficiently and with more complex models of the end goal. We first present a parallelized framework that is interfaced with both the simulation and sampling capabilities of Scenic and the falsification capabilities of VerifAI, reducing the execution time bottleneck inherently present in simulation-based testing. We then present an extension of VerifAI's falsification algorithms to support multi-dimensional objective optimization during sampling, using the concept of rulebooks to specify multiple metrics and a preference ordering over the metrics that can be used to guide the counterexample search process. Lastly, we evaluate the benefits of these extensions with a comprehensive set of experiments written in the Scenic language.},
}

EndNote citation:

%0 Thesis
%A Viswanadha, Kesav 
%T Scalable Techniques for Sampling-Based Falsification of AI-Based Cyber Physical Systems
%I EECS Department, University of California, Berkeley
%D 2021
%8 May 14
%@ UCB/EECS-2021-103
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2021/EECS-2021-103.html
%F Viswanadha:EECS-2021-103