Contour Detection,
Image and Video Segmentation

"When you can measure what you are speaking about and express it in numbers, you know something about it; but when you cannot measure it, when you cannot express it in numbers, your knowledge is of the meager and unsatisfactory kind."

Lord Kelvin

Overview:

The phenomenon of visual grouping was first highlighted by the Gestalt school of visual perception led by Max Wertheimer, nearly a century ago. In computational vision, this ability has been studied as "image segmentation", the partitioning of an image (or video stream) into sets of pixels that correspond to "objects" or parts of objects. This process is based on bottom up cues such as similarity of pixel brightness, color, texture and motion as well as top down input derived from familiar object categories such as faces. Our research is aimed at developing a scientific understanding of grouping, both in the context of human perception and for computer vision. Key contributions include:
  • A large dataset of natural images that have been segmented by human observers. This dataset, available [here], serves as ground truth for learning grouping cues as well as a benchmark for comparing different segmentation and boundary finding algorithms.

  • Computational models of low level cues such as brightness, color, texture and motion, inspired by corresponding mechanisms in primary visual cortex. We use the dataset of human segmented images to calibrate the cues and learn optimum cue combination strategies. The models thus reflect the "ecological statistics" of natural images, which presumably visual animals have evolved to exploit.

  • Algorithms that go from local image cues to globally optimal segmentations. The normalized cut algorithm, based on spectral graph theory, provides a mechanism for going from pairwise pixel affinities based on local cues to partitions that maximize the ratio of affinities within a group to that across groups. By computing oriented gradients on the eigenvectors of the normalized Laplacian, we get a robust signal for marking image contours. We have used Conditional Random Fields as a formalism for combing low-, mid- and high- level cues for grouping and figure-ground discrimination.

Resources:

  • Resources for contour detection and image segmentation, including the Berkeley Segmentation Data Set 500 (BSDS500), are available [here].

  • The Berkeley Segmentation Data Set 300 (BSDS300) is still available [here].

  • Berkeley Grouping and Ecological Statistics page.


Publications:

* "Semantic Contours from Inverse Detectors"
B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik.   ICCV 2011
[pdf] [dataset]
* "Occlusion Boundary Detection and Figure/Ground Assignment from Optical Flow"
P. Sundberg, T. Brox, M. Maire, P. Arbelaez, and J. Malik.   CVPR 2011
[pdf] [dataset part 1] [dataset part 2]
* "Biased Normalized Cuts"
S. Maji, N. Vishnoi, and J. Malik.   CVPR 2011
[pdf]
[Code]
* "Contour Detection and Hierarchical Image Segmentation"
P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik.
TPAMI 2011.
[pdf]
[Resources]
* "Large Displacement Optical Flow: Descriptor Matching in Variational Motion Estimation"
T. Brox, J. Malik.   TPAMI 2011
[pdf] [code]
* "Object segmentation by long term analysis of point trajectories"
T. Brox, and J. Malik.   ECCV 2010
[pdf]
[Dataset]
* "From Contours to Regions: An Empirical Evaluation"
P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik.   CVPR 2009
[pdf]
* "Large Displacement Optical Flow"
T. Brox, C. Bregler, and J. Malik.   CVPR 2009
[pdf]
* "Using Contours to Detect and Localize Junctions in Natural Images"
M. Maire, P. Arbelaez, C. Fowlkes, and J. Malik.   CVPR 2008
[pdf] [slides]
* "Learning Probabilistic Models for Contour Completion in Natural Images"
X. Ren, C. Fowlkes, and J. Malik.   IJCV 2008
[pdf]
* "Cue Integration in Figure/Ground Labeling"
X. Ren, C. Fowlkes, and J. Malik.   NIPS 2005
[pdf]
* "Scale-Invariant Contour Completion using Conditional Random Fields"
X. Ren, C. Fowlkes, and J. Malik.   ICCV 2005
[pdf] [ps.gz]
* "Mid-level Cues Improve Boundary Detection"
X. Ren, C. Fowlkes, and J. Malik.   Technical Report CSD-05-1382
[pdf]
* "How Much Does Globalization Help Segmentation?"
C. Fowlkes, and J. Malik.   Technical Report CSD-04-1340
[pdf]
* "Learning a Classification Model for Segmentation"
X. Ren, and J. Malik.   ICCV 2003
[pdf] [ps.gz]
* "Spectral Grouping Using the Nyström Method"
C. Fowlkes, S. Belongie, F. Chung and J. Malik.   IEEE PAMI, Feb. 2004   [pdf]
* "Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues"
D. Martin, C. Fowlkes, and J. Malik.   to appear in PAMI   [pdf]
[BSDS300]
* "Learning Affinity Functions for Image Segmentation: Combining Patch-based and Gradient-based Approaches"
C. Fowlkes, D. Martin, and J. Malik.   CVPR 2003
[pdf] [ps.gz]
* "Learning to Detect Natural Image Boundaries Using Brightness and Texture"
D. Martin, C. Fowlkes, and J. Malik.   NIPS 2002   [pdf] [ps.gz]
* "A Probabilistic Multi-scale Model for Contour Completion Based on Image Statistics"
X. Ren, J. Malik.   ECCV 2002   [pdf]
* "Spectral Partitioning with Indefinite Kernels using the Nyström Extension"
S. Belongie, C. Fowlkes, F. Chung and J. Malik.   ECCV 2002   [pdf] [ps.gz]
* "Efficient Spatiotemporal Grouping Using the Nyström Method"
C. Fowlkes, S. Belongie, and J. Malik.   CVPR 2001   [pdf] [ps.gz]
* "A Database of Human Segmented Natural Images and its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics"
D. Martin, C. Fowlkes, D. Tal, and J. Malik.   ICCV 2001   [pdf] [ps.gz]
[BSDS300]
* "Contour and Texture Analysis for Image Segmentation"
J. Malik, S. Belongie, T. Leung and J. Shi.   IJCV 2001   [pdf] [ps.gz]
[Results]
* "Normalized Cuts and Image Segmentation"
J. Shi and J. Malik.   IEEE TPAMI, 22(8) Aug. 2000   [pdf] [ps.gz]
* "Textons, Contours and Regions: Cue Combination in Image Segmentation"
J. Malik, S. Belongie, J. Shi and T. Leung.   ICCV 1999   [pdf] [ps.gz]
* "Motion Segmentation and Tracking Using Normalized Cuts"
J. Shi and J. Malik.   ICCV 1998   [pdf] [ps.gz]
* "Contour continuity in region based image segmentation"
T. Leung and J. Malik.   ECCV 1998   [pdf] [ps.gz]
* "Finding Boundaries in Natural Images: A New Method Using Point Desdcriptors and Area Completion"
S. Belongie and J. Malik.   ECCV 1998   [pdf] [ps.gz]
* "Self Inducing Relation Distance and its Application to Image Segmentation"
J. Shi and J. Malik.   ECCV 1998   [pdf] [ps.gz]
* "Normalized Cuts and Image Segmentation"
J. Shi and J. Malik.   CVPR 1997   [ps.gz]



Back to Berkeley Computer Vision page
Questions --> Pablo Arbelaez