Alvin Yuan and Kurt Luther and Markus Krause and Steven P. Dow and Björn Hartmann

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2015-223

December 1, 2015

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2015/EECS-2015-223.pdf

Expert feedback is valuable but hard to obtain for many designers. Online crowds can provide a source of fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted a 2x2 experiment in which student designers received feedback on a visual design artifact from both experts and novices, who produced feedback using either an expert rubric or no rubric. We find that rubrics help novice workers provide feedback that is rated just as valuable as expert feedback. A follow-up analysis on writing style showed that student designers find feedback most helpful when it is emotional, positive, and specific, and that providing a rubric improves the application of these characteristics in feedback. The analysis also finds that expertise only affects style by increasing critique length, but an informal evaluation suggests that experts may instead produce value through providing clearer justifications.

Advisors: Björn Hartmann

---

BibTeX citation:

@mastersthesis{Yuan:EECS-2015-223,
    Author= {Yuan, Alvin and Luther, Kurt and Krause, Markus and Dow, Steven P. and Hartmann, Björn},
    Title= {Worker Expertise and Expert Rubrics in Crowdsourced Design Critique},
    School= {EECS Department, University of California, Berkeley},
    Year= {2015},
    Month= {Dec},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2015/EECS-2015-223.html},
    Number= {UCB/EECS-2015-223},
    Abstract= {Expert feedback is valuable but hard to obtain for many designers. Online crowds can provide a source of fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted a 2x2 experiment in which student designers received feedback on a visual design artifact from both experts and novices, who produced feedback using either an expert rubric or no rubric. We find that rubrics help novice workers provide feedback that is rated just as valuable as expert feedback. A follow-up analysis on writing style showed that student designers find feedback most helpful when it is emotional, positive, and specific, and that providing a rubric improves the application of these characteristics in feedback. The analysis also finds that expertise only affects style by increasing critique length, but an informal evaluation suggests that experts may instead produce value through providing clearer justifications.},
}

EndNote citation:

%0 Thesis
%A Yuan, Alvin 
%A Luther, Kurt 
%A Krause, Markus 
%A Dow, Steven P. 
%A Hartmann, Björn 
%T Worker Expertise and Expert Rubrics in Crowdsourced Design Critique
%I EECS Department, University of California, Berkeley
%D 2015
%8 December 1
%@ UCB/EECS-2015-223
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2015/EECS-2015-223.html
%F Yuan:EECS-2015-223