Wei Wu and Kurt Luther and Amy Pavel and Björn Hartmann and Steven Dow and Maneesh Agrawala

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2013-95

May 17, 2013

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2013/EECS-2013-95.pdf

Crowdsourced labor has successfully tackled a variety of rote tasks like image tagging and transcription, but has difficulty accomplishing complex tasks. Visual design critique is a crucial part of the design process for improving upon existing work, but it is a complex task that requires some implicit knowledge of design concepts on the part of the critiquer. In this report, we propose a model for crowdsourcing visual design critique to enable online workers who lack background knowledge to provide useful feedback on designs. Our model organizes the “crit” around a set of design principles, and asks workers to select relevant statements about design principles and provide freeform comments within the scope of the selected statements. We evaluate two variants of a critiquing system we built based on this model (CrowdCritter) through a study with 73 participants. We found that the integration of a set of design principles allowed novice workers to provide more critiques than freeform feedback alone. We were unable to identify significant differences between the two variants of CrowdCritter in the quality and level of agreement of critiques produced through each interface.

Advisors: Björn Hartmann

---

BibTeX citation:

@mastersthesis{Wu:EECS-2013-95,
    Author= {Wu, Wei and Luther, Kurt and Pavel, Amy and Hartmann, Björn and Dow, Steven and Agrawala, Maneesh},
    Title= {CrowdCritter: Strategies for Crowdsourcing Visual Design Critique},
    School= {EECS Department, University of California, Berkeley},
    Year= {2013},
    Month= {May},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2013/EECS-2013-95.html},
    Number= {UCB/EECS-2013-95},
    Abstract= {Crowdsourced labor has successfully tackled a variety of rote tasks like image tagging and transcription, but has difficulty accomplishing complex tasks. Visual design critique is a crucial part of the design process for improving upon existing work, but it is a complex task that requires some implicit knowledge of design concepts on the part of the critiquer. In this report, we propose a model for crowdsourcing visual design critique to enable online workers who lack background knowledge to provide useful feedback on designs. Our model organizes the “crit” around a set of design principles, and asks workers to select relevant statements about design principles and provide freeform comments within the scope of the selected statements. We evaluate two variants of a critiquing system we built based on this model (CrowdCritter) through a study with 73 participants. We found that the integration of a set of design principles allowed novice workers to provide more critiques than freeform feedback alone. We were unable to identify significant differences between the two variants of CrowdCritter in the quality and level of agreement of critiques produced through each interface.},
}

EndNote citation:

%0 Thesis
%A Wu, Wei 
%A Luther, Kurt 
%A Pavel, Amy 
%A Hartmann, Björn 
%A Dow, Steven 
%A Agrawala, Maneesh 
%T CrowdCritter: Strategies for Crowdsourcing Visual Design Critique
%I EECS Department, University of California, Berkeley
%D 2013
%8 May 17
%@ UCB/EECS-2013-95
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2013/EECS-2013-95.html
%F Wu:EECS-2013-95