Postdoctoral Scholar
University of California, Berkeley
PhD '20 UC Berkeley
Areas of Interest
- Artificial Intelligence
- Graphics
- Human-Computer Interaction
- Computer Vision
Poster
Body2Hands: Learning to Infer 3D Hands from Conversational Gesture Body Dynamics
Abstract
We propose a novel learned deep prior of body motion for 3D hand shape synthesis and estimation in the domain of conversational gestures. Our model builds upon the insight that body motion and hand gestures are strongly correlated in non-verbal communication settings. We formulate the learning of this prior as a prediction task of 3D hand shape over time given body motion input alone. Trained with 3D pose estimations obtained from a large-scale dataset of internet videos, our hand prediction model produces convincing 3D hand gestures, given only the 3D motion of the speaker’s arms as input. We demonstrate the efficacy of our method on hand gesture synthesis from body motion input, and as a strong body prior for single-view image-based 3D hand pose estimation. We demonstrate that our method outperforms previous state-of-the-art approaches and can generalize beyond the monologue based training data to multi-person conversations. Video results are available at http://people.eecs.berkeley.edu/~evonne_ng/projects/body2hands/.
Joint work conducted by Evonne Ng, Hanbyul Joo, Shiry Ginosar, and Trevor Darrell
Bio
Shiry Ginosar is a Computing Innovation Postdoctoral Fellow at UC Berkeley, advised by Jitendra Malik. She completed her Ph.D. in Computer Science at UC Berkeley, under the supervision of Alyosha Efros. Prior to joining the Computer Vision group, she was part of Bjoern Hartmann's Human-Computer Interaction lab at Berkeley. Earlier in her career, she was a Visiting Scholar at the CS Department of Carnegie Mellon University, with Luis von Ahn and Manuel Blum in the field of Human Computation. Between her academic roles, she spent four years at Endeca as a Senior Software Engineer. In her distant past, Shiry trained fighter pilots in F-4 Phantom flight simulators as a Staff Sergeant in the Israeli Air Force. Shiry’s research has been covered by The New Yorker, The Wall Street Journal, and the Washington Post, amongst others. Her work has been featured on PBS NOVA, exhibited at the Israeli Design Museum and is part of the permanent collection of the Deutsches Museum. Her patent-pending research work inspired the founding of a startup. Shiry has been named a Rising Star in EECS, and is a recipient of the NSF Graduate Research Fellowship, the California Legislature Grant for graduate studies, and the Samuel Silver Memorial Scholarship Award for combining intellectual achievement in science and engineering with serious humanistic and cultural interests.
