Poselets and Their Applications in High-Level Computer Vision

Abstract

We address the classic problems of detection and segmentation using a part based detector that operates on a novel part, which we refer to as a poselet. Poselets are tightly clustered in both appearance space (and thus are easy to detect) as well as in configuration space (and thus are helpful for localization and segmentation). We demonstrate poselets are effective for detection, pose extraction, segmentation, action/pose estimation and attribute classification. Poselet construction requires extra annotations beyond the object bounds. To train poselets we have created H3D (Humans in 3D) - a dataset of 1200+ person annotations. The annotations include the joints, the extracted 3D pose, keypoint visibility and region labels. We have also annotated the people in the training and validation sets of PASCAL VOC 2009.

Our poselet classifier achieves state-of-the-art results for the person category on PASCAL VOC 2007, 2008, 2009 and 2010 as well as on our dataset, H3D.

Browse Poselets

You can browse the 150 poselets for the person category.

Results

The following are results as of September 23, 2010 for the Person category of the PASCAL VOC challenges, using Comp 4.

Dataset	Score
VOC 2010	48.5
VOC 2009	48.6
VOC 2008	54.1
VOC 2007	46.9

Papers

Poselets:

Lubomir Bourdev, Jitendra Malik,Poselets: Body Part Detectors Trained Using 3D Human Pose Annotations, ICCV 2009 BibTeX
Lubomir Bourdev, Subhransu Maji, Thomas Brox, Jitendra Malik,Detecting People Using Mutually Consistent Poselet Activations, ECCV 2010 BibTeX

Applications of poselets:

Subhransu Maji, Lubomir Bourdev, Jitendra Malik,Action Recognition from a Distributed Representation of Pose and Appearance, CVPR 2011 BibTeX
Thomas Brox, Lubomir Bourdev, Subhransu Maji, Jitendra Malik,Object Segmentation by Alignment of Poselet Activations to Image Contours, CVPR 2011 BibTeX
Bharath Hariharan, Pablo Arbelaez, Lubomir Bourdev, Subhransu Maji, Jitendra Malik, Semantic Contours from Inverse Detectors, ICCV 2011 BibTeX
Lubomir Bourdev, Subhransu Maji, Jitendra Malik, Describing People: Poselet-Based Approach to Attribute Classification, ICCV 2011BibTeX

Slides

These slides describe the core poselet selection and training algorithm (the ICCV 2009 paper)
Attributes (the ICCV2011 oral)

Videos

Attributes (the ICCV2011 oral)

Code

Below is stand-alone code that takes an image and draws bounding boxes of the people in it in C++ and in Matlab. The Matlab version can also perform interactive visualization of the poselets. Requirements: Matlab + Image Processing toolbox. The code is released with a non-commercial license. The released code and trained detector is similar to the one we used in the PASCAL 2010 competition, which is slightly improved in accuracy (but slower) than our ECCV 2010 paper. The included model was trained for PASCAL2007. The C++ port has nearly identical output and the Matlab code is adapted so it can be used with the Parallel Computing Toolbox (i.e. all globals are removed).

April 2013 release (Matlab). PASCAL 2007 AP 46.41 (47.02 with the old PASCAL measurement method)

April 2013 release (C++). PASCAL 2007 AP 46.34 (46.96 with the old PASCAL measurement method)

Note: If you use WinZip and Matlab reports that your file is corrupt, please try WinRAR. If you need an older release please let is know.

Datasets

H3D and the associated Matlab toolbox.
Keypoint annotations for the people in PASCAL 2009 training and validation images.
3D pose estimation (see project page for more)
Keypoint annotations for the PASCAL 2010 action dataset (see project page for more)
Attributes of people dataset.
Dataset of keypoints and foreground annotations for all categories of PASCAL 2011

H3D Annotation tool

The Java3D tool that we used to create H3D and a video tutorial are available here. There are no license restrictions on using the tool for your own annotations.

Contact us

For comments or questions about poselets please email Lubomir Bourdev lubomir.bourdev -at- gmail.com.