Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg and Pieter Abbeel

EECS Department
University of California, Berkeley
Technical Report No. UCB/EECS-2020-190
December 1, 2020

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-190.pdf

Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach for learning visuomotor skills.

Advisor: Pieter Abbeel


BibTeX citation:

@mastersthesis{Zhang:EECS-2020-190,
    Author = {Zhang, Tianhao and McCarthy, Zoe and Jow, Owen and Lee, Dennis and Chen, Xi and Goldberg, Ken and Abbeel, Pieter},
    Title = {Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation},
    School = {EECS Department, University of California, Berkeley},
    Year = {2020},
    Month = {Dec},
    URL = {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-190.html},
    Number = {UCB/EECS-2020-190},
    Abstract = {Imitation learning is a powerful paradigm for robot skill acquisition. However, obtaining demonstrations suitable for learning a policy that maps from raw pixels to actions can be challenging. In this paper we describe how consumer-grade Virtual Reality headsets and hand tracking hardware can be used to naturally teleoperate robots to perform complex tasks. We also describe how imitation learning can learn deep neural network policies (mapping from pixels to actions) that can acquire the demonstrated skills. Our experiments showcase the effectiveness of our approach for learning visuomotor skills.}
}

EndNote citation:

%0 Thesis
%A Zhang, Tianhao
%A McCarthy, Zoe
%A Jow, Owen
%A Lee, Dennis
%A Chen, Xi
%A Goldberg, Ken
%A Abbeel, Pieter
%T Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation
%I EECS Department, University of California, Berkeley
%D 2020
%8 December 1
%@ UCB/EECS-2020-190
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2020/EECS-2020-190.html
%F Zhang:EECS-2020-190