A Scalable Synchronous System for Robust Real Time Human-Machine Collaboration

Christopher Powers

EECS Department
University of California, Berkeley
Technical Report No. UCB/EECS-2020-87
May 28, 2020

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-87.pdf

Large labeled datasets are vital for applying supervised learning to computer vision tasks, but making these datasets takes time and effort. Human-machine collaboration has the potential to mitigate this cost. This work introduces a general-purpose framework for human-to-human and human-machine collaboration on image data. We show that by treating machine learning models as virtual users, multi-user synchronization can support versatile human-machine interaction; in other words, all you need is sync. In order to achieve synchronization behavior that seems correct to users, while also maintaining real-time editing speeds and supporting undo-redo, we adapt operational transformation [8] to the image labeling context. An open- source implementation of the collaboration system is presented as an in-progress addition to Scalabel, an annotation tool for visual data that was used to create the BDD100K dataset [24]. Finally, we give an example of how the collaboration feature can improve the labeling process by integrating Polygon-RNN++ [2] with Scalabel.

Advisor: Trevor Darrell


BibTeX citation:

@mastersthesis{Powers:EECS-2020-87,
    Author = {Powers, Christopher},
    Title = {A Scalable Synchronous System for Robust Real Time Human-Machine Collaboration},
    School = {EECS Department, University of California, Berkeley},
    Year = {2020},
    Month = {May},
    URL = {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-87.html},
    Number = {UCB/EECS-2020-87},
    Abstract = {Large labeled datasets are vital for applying supervised learning to computer vision tasks, but making these datasets takes time and effort. Human-machine collaboration has the potential to mitigate this cost. This work introduces a general-purpose framework for human-to-human and human-machine collaboration on image data. We show that by treating machine learning models as virtual users, multi-user synchronization can support versatile human-machine interaction; in other words, all you need is sync. In order to achieve synchronization behavior that seems correct to users, while also maintaining real-time editing speeds and supporting undo-redo, we adapt operational transformation [8] to the image labeling context. An open- source implementation of the collaboration system is presented as an in-progress addition to Scalabel, an annotation tool for visual data that was used to create the BDD100K dataset [24]. Finally, we give an example of how the collaboration feature can improve the labeling process by integrating Polygon-RNN++ [2] with Scalabel.}
}

EndNote citation:

%0 Thesis
%A Powers, Christopher
%T A Scalable Synchronous System for Robust Real Time Human-Machine Collaboration
%I EECS Department, University of California, Berkeley
%D 2020
%8 May 28
%@ UCB/EECS-2020-87
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-87.html
%F Powers:EECS-2020-87