\documentclass[12pt]{article}
\usepackage{fullpage,psfig}

\newcommand{\lecturetitle}[4]{
  \noindent
  CS 294-2, Grouping and Recognition (Prof. Jitendra Malik) \hfill #3\\
  Lecture \##1 (#2) \hfill {\bf DRAFT} Notes by #4\\[-.05in]
  \mbox{}\hrulefill \mbox{}\\}

\newcommand{\R}{{\bf R}}
% \setlength{\parindent}{0mm}

\begin{document}

% MODIFY THE FOLLOWING TO INCLUDE THE LECTURE #, TOPIC, DATE, and YOUR NAME
\lecturetitle{8}{Grouping by motion}{Sept 15, 1999}{Omid Shakernia}

% INCLUDE AN OUTLINE OF THE CLASS TOPICS

This lecture was given by Yair Weiss.
\begin{itemize} 
  \item Wertheimer principles of grouping
  \item Grouping based on motion
  \item Horn \& Schunk algorithm
  \item Layered approach to grouping
  \item Conclusions
\end{itemize}


% THE NOTES WILL GO HERE

\section{Wertheimer's principles of grouping}
Recall from Wertheimer's principles of grouping,\footnote{See the paper 
``Laws of organization in perceptual forms''  in \\
{\tt http://www.yorku.ca/dept/psych/classics/Wertheimer/Forms/forms.htm}}
the principle factors which lead to grouping include:
\begin{itemize}
\item proximity 
\item similarity (brightness, color, shape, texture, motion, etc.)
\item good continuation of boundary contours
\item closure
\item symmetry and parallelism
\item familiar configuration
\end{itemize}

\section{Grouping based on motion}

Grouping based on motion is difficult because of the 
{\em aperture problem}.  Figure~\ref{fig:aper} gives a graphical
representation of the aperture problem.  When we only see the local
motion  of a contour, we can only see the component of the motion
orthogonal to the local contour.  That is, if we see the edge moving to the
right, we have no way of knowing whether the motion of the edge has
any vertical component.

\begin{figure}[h]
    \centerline{\psfig{figure=aperture.eps}}
    \caption{Graphical deptiction of the aperture problem}
    \label{fig:aper}
\end{figure}

The aperture problem can be described mathematically by the equation
\begin{eqnarray}
  \label{eq:aper}
  I_x v_x + I_y v_y + I_t = 0
\end{eqnarray}

Given an image brightness pattern sequenced in time, it is a
good approximation that locally the overall brightness stays constant
over small intervals of time.  Denoting the image sequence as
$I(x,y,t)$, the approximation says that
\begin{eqnarray*}
  \label{eq:bright}
  0 &=& \frac{d}{dt}I(x,y,t)\\
  &=& \frac{\partial I}{\partial x} \frac{dx}{dt} + \frac{\partial
  I}{\partial y}  \frac{dx}{dt} + \frac{\partial I}{\partial t} \\
  &=:& I_x v_x + I_y v_y + I_t 
\end{eqnarray*}
Equation~(\ref{eq:aper}) comes directly from the constant brightness
condition.
$(I_x, I_y) \in \R^2$ is the gradient of the image brightness 
and $(v_x, v_y) \in \R^2$ is the velocity of the point $(x,y)$.  In the
example illustrated in Figure~\ref{fig:aper} the brightness gradient
of the  center pixel is porportional to  $(I_x, I_y) = (1,0)$.
Notice that for this image brightness gradient,
equation~(\ref{eq:aper}) does not constrain $v_y$.  That means for a
given horizontal motion $v_x$ any vertical component of motion is
consistent with the constraint~(\ref{eq:aper}).
The aperture problem was demonstrated by a video of the two squares
moving diagonally, where we only had local views of edges of the squares. 

\section{Horn \& Schunk algorithm}

Horn \& Schunk (1982) proposed an algorithm to compute the image
velocity vector field by combining the local velocity measurements
with a smoothness cost.  The algorithm incorporates the constraint in 
equation~(\ref{eq:aper}) with a cost for non-smooth vector fields,
thus attempting to regularize the aperture problem by combining local
measurements. The cost function considered in the Horn \& Schunk
algorithm is: 

\begin{eqnarray}
  J(v) &=& \sum_{(x,y)\in I} (I_x v_x + I_y v_y + I_t)^2 + \label{eq:horn1}\\
  && \gamma_1  \|v(x,y) - v(x-1,y)\|^2 +  \gamma_2  \|v(x,y) - v(x,y-1)\|^2 \label{eq:horn2}
\end{eqnarray}
Then a  probability distribution on velocity vector fields is given by
\begin{eqnarray}
  \label{eq:prob}
  p(v) = \frac{1}{\lambda} e^{-J(v)}
\end{eqnarray}
Equation~(\ref{eq:prob}) is a special case of a Markov Random Field
(MRF) since the velocity field only depends on its nearest neighbors.  
In the cost function $J(v)$, the term~(\ref{eq:horn1}) can be thought
of as the likelihood term, and the term~(\ref{eq:horn2}) which is a
smoothness cost can be thought of as the prior.

There are several criticisms of the Horn \& Schunk algorithm including
\begin{itemize}
\item Is this the right prior to use?
\item There's no notion of grouping
\item Fails if there is more than one moving object
\end{itemize}

In the cases that there is more than one moving object, the smoothness
constraint would actually be detrimental.  In such cases, the
constraint should really be a piecewise smoothness constraint, with possible
discontinuities at the boundaries of the moving objects.  One approach
to incorporate this idea would be to add a line process to the cost function.

However, there is evidence to suggest that Horn \& Schunk's algorithm
is not the 
right model for the way the human visual system works.  To demonstrate
this, there was a demo video of two ellipses rotating.  The thin
ellips looked as if it were one rigid body rotating, but the fat
ellipse looked as if it were deforming.  Figure~\ref{fig:ellipse}
should remind the reader of the video.

\begin{figure}[h]
  \centerline{\psfig{figure=ellipse.eps}}
    \caption{Two ellipses rotating}
    \label{fig:ellipse}
\end{figure}

Also, there are experiments that show that people are willing to group
objects across motion discontinuities.  For example, there was a demo
video of an ellipse with 4 satallite points that were  rotating
together on top of a static textured background.  The fact that there
was static texture between the ellipse and the satellite points does
not stop people from grouping the ellipse and the satellite points.

To paraphrase a quote the Italian Gestalt psychologist Matteli (since
I didn't catch the exact wording):  
\begin{quote}
  We don't measure the motion of points, but the motion of the group to which
  the points belong. 
\end{quote}

\section{Layered Approach to Grouping}

An approach to explaining why humans group objects across motion
discontinuities is the approach of requiring smoothness in layers.
Within each layer, there should be some smoothness in the velocity
vector field.  This layered approach would explain the fact people are
willing to group object across motion discontinuities.

For illustration purposes, we were shown an example that looked
roughly like Figure~\ref{fig:layers}.
\begin{figure}[h]
  \centerline{\psfig{figure=layers.eps}}
  \caption{Example of smoothness in layers}
  \label{fig:layers}
\end{figure}

Next we briefly reviewed the Expectation Maximization algorithm.
Recall that the EM algorithm is an example 
of mixture estimation where the hidden variables we are trying to
estimate are the group to which the data points belong. We were shown
a demonstration of fitting two lines to a given data set.  We also saw
a demo of curve fitting for a group of lines


\subsection{Parametric vs. Non-parametric motion estimation}

Line estimation in EM is easy because we are only  trying to
estimate 2 parameters (slope, etc).    People have also tried to do
parametric motion estimation by assuming that each layer undergoes an
affine motion.  That is, assuming that for each layer, the motion can
be parameterized by:
\begin{eqnarray}
  \label{eq:affine}
  \left( 
    \begin{array}{c}
      v_x \\ v_y
    \end{array}
    \right) = \left(
      \begin{array}{cc}
        a & b \\
        d & e
      \end{array} \right)
    \left(
      \begin{array}{c}
        x \\ y
      \end{array} \right)
    + \left(
      \begin{array}{c}
        c \\ f
      \end{array} \right)
\end{eqnarray}
and then trying to estimate $a,b,c,d,e,f$.  The referrence for
parametric motion estimation is Ayer \& Sawhney. However an affine
motion model is not a good approximation if the scene is not roughly
planar.  Weiss has studied the case of nonparametric motion estimation.

The model selection problem (figuring out the number of groups
necessary for the mixture estimation) is in general very hard 
to solve.  There have been many different approaches to solve the
model selection problem.  One approach is to find the {\em smallest\/}
number of layers consistent with the motion.

\section{Conclusions}

The main message of this lecture was that grouping based on common
fate is more difficult and is different than grouping based on
similarity.  Also grouping based on motion can not be done locally.

\end{document}

