Visually-Grounded Bayesian Word Learning

Yangqing Jia, Joshua Abbott, Joseph Austerweil, Thomas Griffiths and Trevor Darrell

EECS Department
University of California, Berkeley
Technical Report No. UCB/EECS-2012-202
October 17, 2012

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-202.pdf

Learning the meaning of a novel noun from a few labeled objects is one of the simplest aspects of learning a language, but approximating human performance on this task is still a significant challenge for current machine learning systems. Current methods typically fail to find the appropriate level of generalization in a concept hierarchy for a given visual stimulus. Recent work in cognitive science on Bayesian models of word learning partially addresses this challenge, but it assumes that the labels of objects are given (hence no object recognition) and it has only been evaluated in small domains. We present a system for learning nouns directly from images, using probabilistic predictions generated by visual classifiers as the input to Bayesian word learning, and compare this system to human performance in an automated, large-scale experiment. The system captures a significant proportion of the variance in human responses. Combining the uncertain outputs of the visual classifiers with the ability to identify an appropriate level of abstraction that comes from Bayesian word learning allows the system to outperform alternatives that either cannot deal with visual stimuli or use a more conventional computer vision approach.


BibTeX citation:

@techreport{Jia:EECS-2012-202,
    Author = {Jia, Yangqing and Abbott, Joshua and Austerweil, Joseph and Griffiths, Thomas and Darrell, Trevor},
    Title = {Visually-Grounded Bayesian Word Learning},
    Institution = {EECS Department, University of California, Berkeley},
    Year = {2012},
    Month = {Oct},
    URL = {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-202.html},
    Number = {UCB/EECS-2012-202},
    Abstract = {Learning the meaning of a novel noun from a few labeled objects is one of the simplest aspects of learning a language, but approximating human performance on this task is still a significant challenge for current machine learning systems. Current methods typically fail to find the appropriate level of generalization in a concept hierarchy for a given visual stimulus. Recent work in cognitive science on Bayesian models of word learning partially addresses this challenge, but it assumes that the labels of objects are given (hence no object recognition) and it has only been evaluated in small domains. We present a system for learning nouns directly from images, using probabilistic predictions generated by visual classifiers as the input to Bayesian word learning, and compare this system to human performance in an automated, large-scale experiment. The system captures a significant proportion of the variance in human responses. Combining the uncertain outputs of the visual classifiers with the ability to identify an appropriate level of abstraction that comes from Bayesian word learning allows the system to outperform alternatives that either cannot deal with visual stimuli or use a more conventional computer vision approach.}
}

EndNote citation:

%0 Report
%A Jia, Yangqing
%A Abbott, Joshua
%A Austerweil, Joseph
%A Griffiths, Thomas
%A Darrell, Trevor
%T Visually-Grounded Bayesian Word Learning
%I EECS Department, University of California, Berkeley
%D 2012
%8 October 17
%@ UCB/EECS-2012-202
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-202.html
%F Jia:EECS-2012-202