Exploratory model analysis for machine learning
Biye Jiang
EECS Department, University of California, Berkeley
Technical Report No. UCB/EECS-2018-34
May 9, 2018
http://www2.eecs.berkeley.edu/Pubs/TechRpts/2018/EECS-2018-34.pdf
Machine learning is growing in importance in many different fields. However, it is still very hard for users to tune hyper-parameters when optimizing their models, or perform a comprehensive and interpretable diagnosis for complex models like deep neural nets. Existing developer tool like TensorBoard only provides limited functionality which usually visualizes model statistics based on metrics predefined before the training starts. Almost nothing can be adjusted during the training. But the real model optimization and diagnosis procedures actually involve lots of interaction and continuous experiments. To tackle those challenges, we developed a framework which allows users to perform exploratory model analysis based on the hardware accelerated machine learning toolkit BIDMach. Our system is unique that we allow users to interactively add visualizations and adjust hyper-parameters during training. Also, we use Monte Carlo style algorithms to allow users to explore the entire model space under different user preferences rather than only reaching the local optimums.
We demonstrate the usage of our system in several real-world applications. For problems like advertisement optimization or clustering where multiple optimization objectives exist, users can incorporate secondary criteria into the model-generation process and make trade-offs in an interactive way. For deep convolution neural net diagnosis, users can use our LDAM (Langevin Dynamic Activation Maximization) algorithm to systematically explore the images that stimulate a given neuron. We conduct experiments and user studies on several public datasets to show how users from different background can benefit from our tools.
Advisors: John F. Canny
BibTeX citation:
@phdthesis{Jiang:EECS-2018-34,
Author= {Jiang, Biye},
Title= {Exploratory model analysis for machine learning},
School= {EECS Department, University of California, Berkeley},
Year= {2018},
Month= {May},
Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2018/EECS-2018-34.html},
Number= {UCB/EECS-2018-34},
Abstract= {Machine learning is growing in importance in many different fields. However, it is still very hard for users to tune hyper-parameters when optimizing their models, or perform a comprehensive and interpretable diagnosis for complex models like deep neural nets. Existing developer tool like TensorBoard only provides limited functionality which usually visualizes model statistics based on metrics predefined before the training starts. Almost nothing can be adjusted during the training. But the real model optimization and diagnosis procedures actually involve lots of interaction and continuous experiments. To tackle those challenges, we developed a framework which allows users to perform exploratory model analysis based on the hardware accelerated machine learning toolkit BIDMach. Our system is unique that we allow users to interactively add visualizations and adjust hyper-parameters during training. Also, we use Monte Carlo style algorithms to allow users to explore the entire model space under different user preferences rather than only reaching the local optimums.
We demonstrate the usage of our system in several real-world applications. For problems like advertisement optimization or clustering where multiple optimization objectives exist, users can incorporate secondary criteria into the model-generation process and make trade-offs in an interactive way. For deep convolution neural net diagnosis, users can use our LDAM (Langevin Dynamic Activation Maximization) algorithm to systematically explore the images that stimulate a given neuron. We conduct experiments and user studies on several public datasets to show how users from different background can benefit from our tools.},
}
EndNote citation:
%0 Thesis %A Jiang, Biye %T Exploratory model analysis for machine learning %I EECS Department, University of California, Berkeley %D 2018 %8 May 9 %@ UCB/EECS-2018-34 %U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2018/EECS-2018-34.html %F Jiang:EECS-2018-34