Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings

Baiyu Chen, Sergio Escalera, Isabelle Guyon, Victor Ponce-Lopez, Nihar Shah and Marc Oliu Simon

EECS Department
University of California, Berkeley
Technical Report No. UCB/EECS-2017-194
December 5, 2017

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-194.pdf

We address the problem of calibration of workers whose task is to label patterns with continuous variables. An example would be labeling pictures of people with apparent age. Worker bias is particularly difficult to evaluate and correct when many workers contribute just a few labels, a situation arising typically when labeling is crowd-sourced. In the scenario of labeling short videos of people facing a camera with personality traits, we evaluate the feasibility of the pairwise ranking method to alleviate bias problems. Workers are exposed to pairs of patterns at a time and must just rank them with respect to the (presumed) variable level. The variable levels are reconstructed by fitting a Bradley-Terry-Luce model with maximum likelihood. This method may at first sight, seem prohibitively expensive because for N videos, p=N(N-1)/2 pairs must be potentially processed by workers rather that N videos. However, by performing extensive simulations, we determine an empirical law for the scaling of the number of pairs needed as a function of the number of videos in order to achieve a given accuracy of score reconstruction and show that the pairwise method is very affordable.

Advisor: Alexei (Alyosha) Efros


BibTeX citation:

@mastersthesis{Chen:EECS-2017-194,
    Author = {Chen, Baiyu and Escalera, Sergio and Guyon, Isabelle and Ponce-Lopez, Victor and Shah, Nihar and Oliu Simon, Marc},
    Title = {Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings},
    School = {EECS Department, University of California, Berkeley},
    Year = {2017},
    Month = {Dec},
    URL = {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-194.html},
    Number = {UCB/EECS-2017-194},
    Abstract = {We address the problem of calibration of workers whose task is to label patterns with continuous variables. An example would be labeling pictures of people with apparent age. Worker bias is particularly difficult to evaluate and correct when many workers contribute just a few labels, a situation arising typically when labeling is crowd-sourced. In the scenario of labeling short videos of people facing a camera with personality traits, we evaluate the feasibility of the pairwise ranking method to alleviate bias problems. Workers are exposed to pairs of patterns at a time and must just rank them with respect to the (presumed) variable level. The variable levels are reconstructed by fitting a Bradley-Terry-Luce model with maximum likelihood. This method may at first sight, seem prohibitively expensive because for N videos, p=N(N-1)/2 pairs must be potentially processed by workers rather that N videos. However, by performing extensive simulations, we determine an empirical law for the scaling of the number of pairs needed as a function of the number of videos in order to achieve a given accuracy of score reconstruction and show that the pairwise method is very affordable.}
}

EndNote citation:

%0 Thesis
%A Chen, Baiyu
%A Escalera, Sergio
%A Guyon, Isabelle
%A Ponce-Lopez, Victor
%A Shah, Nihar
%A Oliu Simon, Marc
%T Overcoming Calibration Problems in Pattern Labeling with Pairwise Ratings
%I EECS Department, University of California, Berkeley
%D 2017
%8 December 5
%@ UCB/EECS-2017-194
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-194.html
%F Chen:EECS-2017-194