Anik Gupta

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2024-100

May 13, 2024

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/EECS-2024-100.pdf

This paper introduces a novel methodology for automating the animation of macaque monkey avatars by predicting blendshapes from live video streams and recordings. Our approach leverages advancements in computer vision, particularly in the generation of multifaceted synthetic datasets using stable diffusion techniques, tailored to a macaque avatar.

Our method revolves around training a model to directly predict blendshapes from images. We create a synthetic dataset that contains fully accurate ground-truth avatar renders, depth images, blendshapes, camera parameters, and avatar vertices. Importantly, no manual annotations are needed on real imagery for training, and the entire dataset creation process is automated.

Additionally, we utilize an off-the-shelf ResNet model for regressing blendshapes from our generated macaque avatar image datasets. Remarkably, due to the comprehensive nature of our dataset, the training process requires no special modifications.

Experimental results demonstrate the effectiveness of our approach in accurately animating macaque monkey avatars and capturing a diverse range of facial expressions. By providing researchers with an automated and cost-effective tool for avatar animation, our paper contributes to applications in various research domains, especially neuroscience.

Advisors: Jitendra Malik

---

BibTeX citation:

@mastersthesis{Gupta:EECS-2024-100,
    Author= {Gupta, Anik},
    Title= {Markerless Macaque Face Reconstruction Using Synthetic Data},
    School= {EECS Department, University of California, Berkeley},
    Year= {2024},
    Month= {May},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/EECS-2024-100.html},
    Number= {UCB/EECS-2024-100},
    Abstract= {This paper introduces a novel methodology for automating the animation of macaque monkey avatars by predicting blendshapes from live video streams and recordings. Our approach leverages advancements in computer vision, particularly in the generation of multifaceted synthetic datasets using stable diffusion techniques, tailored to a macaque avatar.

Our method revolves around training a model to directly predict blendshapes from images. We create a synthetic dataset that contains fully accurate ground-truth avatar renders, depth images, blendshapes, camera parameters, and avatar vertices. Importantly, no manual annotations are needed on real imagery for training, and the entire dataset creation process is automated.

Additionally, we utilize an off-the-shelf ResNet model for regressing blendshapes from our generated macaque avatar image datasets. Remarkably, due to the comprehensive nature of our dataset, the training process requires no special modifications.

Experimental results demonstrate the effectiveness of our approach in accurately animating macaque monkey avatars and capturing a diverse range of facial expressions. By providing researchers with an automated and cost-effective tool for avatar animation, our paper contributes to applications in various research domains, especially neuroscience.},
}

EndNote citation:

%0 Thesis
%A Gupta, Anik 
%T Markerless Macaque Face Reconstruction Using Synthetic Data
%I EECS Department, University of California, Berkeley
%D 2024
%8 May 13
%@ UCB/EECS-2024-100
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2024/EECS-2024-100.html
%F Gupta:EECS-2024-100