Anup Hiremath
EECS Department
University of California, Berkeley
Technical Report No. UCB/EECS-2022-206
August 12, 2022
http://www2.eecs.berkeley.edu/Pubs/TechRpts/2022/EECS-2022-206.pdf
In this report, we propose a multimodal unsupervised video learning algorithm designed to incorporate information from any number of modalities present in the data. We cooperatively train a network corresponding to each modality: at each stage of training, one of these networks is selected to be trained using the output of the other networks. To verify our algorithm, we train a model using RGB, optical flow, and audio. We then evaluate the effectiveness of our unsupervised learning model by performing action classification and nearest neighbor retrieval on a supervised dataset. We compare this triple modality model to contrastive learning models using one or two modalities, and find using all three modalities in tandem provides a 1.5% improvement in UCF101 classification accuracy, a 1.4% improvement in R@1 retrieval recall, a 3.5% improvement in R@5 retrieval recall, and a 2.4% improvement in R@10 retrieval recall as compared to using only RGB and optical flow, demonstrating the merit of utilizing as many modalities as possible in a cooperative learning model.
Advisor: Avideh Zakhor
BibTeX citation:
@mastersthesis{Hiremath:EECS-2022-206, Author = {Hiremath, Anup}, Editor = {Zakhor, Avideh and Friedland, Gerald}, Title = {Multimodal Contrastive Learning for Unsupervised Video Representation Learning}, School = {EECS Department, University of California, Berkeley}, Year = {2022}, Month = {Aug}, URL = {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2022/EECS-2022-206.html}, Number = {UCB/EECS-2022-206}, Abstract = {In this report, we propose a multimodal unsupervised video learning algorithm designed to incorporate information from any number of modalities present in the data. We cooperatively train a network corresponding to each modality: at each stage of training, one of these networks is selected to be trained using the output of the other networks. To verify our algorithm, we train a model using RGB, optical flow, and audio. We then evaluate the effectiveness of our unsupervised learning model by performing action classification and nearest neighbor retrieval on a supervised dataset. We compare this triple modality model to contrastive learning models using one or two modalities, and find using all three modalities in tandem provides a 1.5% improvement in UCF101 classification accuracy, a 1.4% improvement in R@1 retrieval recall, a 3.5% improvement in R@5 retrieval recall, and a 2.4% improvement in R@10 retrieval recall as compared to using only RGB and optical flow, demonstrating the merit of utilizing as many modalities as possible in a cooperative learning model.} }
EndNote citation:
%0 Thesis %A Hiremath, Anup %E Zakhor, Avideh %E Friedland, Gerald %T Multimodal Contrastive Learning for Unsupervised Video Representation Learning %I EECS Department, University of California, Berkeley %D 2022 %8 August 12 %@ UCB/EECS-2022-206 %U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2022/EECS-2022-206.html %F Hiremath:EECS-2022-206