\documentstyle[12pt,fullpage,psfig]{article}

\newcommand{\lecturetitle}[4]{
  \noindent
  CS 280, Computer Vision (Prof. Jitendra Malik) \hfill #3\\
  Lecture \##1 (#2) \hfill {\bf DRAFT} Notes by #4\\[-.05in]
  \mbox{}\hrulefill \mbox{}\\}

\begin{document}

% MODIFY THE FOLLOWING TO INCLUDE THE LECTURE #, TOPIC, DATE, and YOUR NAME
\lecturetitle{7}{Optical Flow (Continued)}{Feb 10, 1999}{Stephen David}

% INCLUDE AN OUTLINE OF THE CLASS TOPICS
\begin{itemize} 
  \item Spherical Projection of Optical Flow
  \subitem Focus of expansion
  \subitem Scale factor ambiguity
  \subitem Parallax
  \item Inverse Problem: Motion from Flow Field
  \subitem Least squared error fitting
  \subitem Error for motion estimation
  \subitem Gradient of a function
  
\end{itemize}

% THE NOTES WILL GO HERE

\section{Spherical Projection of Optical Flow}

In the previous lecture, we derived an equation for the spherical
projection of optical flow for a viewer moving with translation 
$\vec{t}$ and rotation $\vec{\omega}$:
\begin{equation}
\dot{q}=-\frac{1}{||\vec{X}||} 
(\vec{t}-(\vec{t} \ldotp \vec{q})\vec{q}) - \vec{\omega} \wedge \vec{q}
\label{flow1}
\end{equation}
where $\vec{x}$ is the position of a point in space and $\vec{q}$ is its
projection onto the viewing sphere.

We can write this equation in a simpler way if we make some substitutions.  Recall from earlier lectures that we defined $\hat{q}$ to be the
skew-symmetric matrix which calculates cross product by $\vec{q}$
({\it{i.e.}, $\hat{q}\vec{x}=\vec{q} \wedge \vec{x}$}).  If we let 
$||\vec{q}|| = 1$, then we also found that
\[\hat{q}^2 = \vec{q} \vec{q}^T - I.\]

Additionally, we define {\bf nearness}, $\lambda$, to be
$$\lambda = \frac{1}{||\vec{X}||} = \frac{1}{r},$$
where $r$ is the distance to the point $\vec{X}$.

Substituting these values into equation \ref{flow1}, we get:
\begin{equation}
\dot{q}=\lambda \hat{q}^2 \vec{t} + \hat{q}\vec{\omega}.
\label{flow2}
\end{equation}

\subsection{Focus of expansion}

Suppose we observe the special case of a system undergoing pure translation 
($\vec \omega=0$).  In that case,
$$\dot{q}=\lambda \hat{q}^2 \vec{t}=\lambda \vec q \wedge (\vec q \wedge
\vec t).$$
Notice that $\dot q=0$ when $\vec q || \vec t$.  This value of $\vec q$ is
called the {\bf focus of expansion}, and it indicates the direction of
pure translational motion.

Suppose the camera is undergoing pure rotation ($\vec t=0$).  In that
case, $ \dot q=\vec q \wedge \vec \omega,$ and $\dot q=0$ at points
parallel to the access of rotation ({\it i.e.}, $\vec q || \vec
\omega$).

\subsection{Scale factor ambiguity}

Because projection from a
3-d world onto a 2-d view sphere necessarily loses some information
about the scene, certain features of the scene can not be recovered from
optical flow alone.  More specifically, we can measure the product, 
$\lambda ||\vec t||$, but we can not measure the ratio of $\lambda$ to 
$||\vec t||$.  This effect, {\bf scale factor ambiguity}, gives rise to
the phenomenon that objects moving slowly at a short distance appear
similar to object far away moving in the same direction at a higher
speed.

\subsection{Parallax}

\begin{figure}[htb]
\centerline{\psfig{file=parallax.eps,height=2in}}
\caption{Example of parallax.  Points near the boundary of $O_1$ and $O_2$
show different optical flows because they lie at different distances.}
\label{fparallax}
\end{figure}

Optical flow provides useful information about depth differences between two 
adjacent objects in a scene.  Let $\vec q_-$ and $\vec q_+$ be two points 
in the image near the edge of two objects, $O_1$ and $O_2$ (see figure 
\ref{fparallax}), 
at different depths, $\lambda_-$ and $\lambda_+$ in a scene.  The
corresponding optical flow for each of these points are
\[\dot{q}_-=\lambda_- \hat{q}^2_- \vec{t} + \hat{q}_-\vec{\omega}.\]
and
\[\dot{q}_+=\lambda_+ \hat{q}^2_+ \vec{t} + \hat{q}_+\vec{\omega}\]
At points near the boundary of $O_1$ and $O_2$ in the image, 
$\vec q = \vec q_- \approx \vec q_+$.  We define {\bf parallax} as the 
difference between the two flows:
\[\vec q_- - \vec q_+ = (\lambda_- - \lambda_+) \hat{q}^2\vec t + 0\]
The rotational component of flow cancels out, but the translational
component remains, due to the different values of $\lambda$ near the
edge.

\section{Inverse Problem: Motion from Flow Field}

Suppose we can measure optical flow at a number of points in the visual
field.  (This assumes that we have solved the correspondence problem
required to identify sets of two points in two different images as the same, but
that's another issue reserved for later in the semester.)  We wish to
use optical flow data to calculate how a scene is translating and rotating.  
Referring to equation \ref{flow2}, we have values for $\vec q_i$ and 
$\dot q_i$ where $i=1 \ldots N$, and we would like to find values for 
$\vec t$, $\vec \omega$, and $\lambda_i$.  Note that the values of 
$\vec t$ and $\vec \omega$ are constant for the entire image, but 
$\lambda_i$ can change with each point $\vec q_i$.

Since $\vec t$ and $\vec \omega$ are both 3-d vectors and each $\lambda_i$
is a scalar, we are looking for $6+N$ unknowns.  Due to the scale factor
ambiguity, however, we can only hope to resolve $5+N$ of those unknowns.
Each $\vec q_i$ is a 2-d vector, giving two equations.  In theory, we need 
a minumum of $N=5$ points to give us 10 equations and 10 unknowns to solve
the problem exactly.  Because of noise, however, it's generally better
to have a value of $N$ significantly larger than 5.

\subsection{Least squared error fitting}

We solve the motion from flow problem using a least square error method.
The idea is that because there is noise in the system, we aren't likely
to find a solution that would give us the same values of flow back
if we were to plug $\vec q_i$ and our calculated values for $\vec t$, 
$\vec \omega$, and $\lambda_i$ back into equation \ref{flow2}.  Instead,
we try to minimize the square of the error between our approximation of
flow, $\dot q_i'$ and the actual value of $\dot q_i$.

How do we measure error?  Suppose we have a some function $y=f(x)$ for
which we've measured $N$ values of $(x_i,y_i)$.  We approximate $f$ as a
linear function,
\[f(x)=a_1x + a_2.\]
The error of our approximation is:
\begin{equation}
MSE = \frac{1}{N}\sum_{i=1}^{N} (y_i - a_1x_i - a_2)^2
\label{MSE}
\end{equation}

\subsection{Error for motion estimation}

We use the notation of Koenderink \& Van Doorn ("Facts on Optic Flow",
{\it Biol. Cybernetics}. 56, 247-254, 1987) for our MSE calculation. 
Let $d_i=q_i$, $p_i=\dot q_i$, and $\mu_i=\lambda_i$.  Furthermore,
assume $||\vec t||=1$ to account for scale factor ambiguity.  We
continue to use $\vec t$ and $\vec \omega$ for our motion vectors.

We use equation \ref{MSE} to calculate error:
\begin{equation}
E=\frac{1}{N}\sum_{i=1}^{N} ||\vec p_i-\vec p_i'||^2
\label{flowMSE}
\end{equation}

Where we rewrite equation \ref{flow2}: 
$\vec p_i'=\mu_i \hat{d_i}^2\vec t + \hat{d_i}\vec \omega$.

\subsection{Gradient of a function}

Now that we've got the MSE, how do we find the $\vec t$, $\vec \omega$,
and $\mu_i$ to minimize it?  As we learned in calculus, we know that a
function will reach its local minima and maxima at points where the
derivative is 0.

To find the derivative of the MSE with respect to its vector parameters,
we must take the {\bf gradient}.  Recall the definition of gradient for
$f: \Re^3 \rightarrow \Re$:
\[\nabla f= \left[\matrix{\frac{\partial f}{\partial x}\cr
\frac{\partial f}{\partial y}\cr
\frac{\partial f}{\partial z}}\right].\]

So, for example, if $f=2x+3xy+z^2$,
\[\nabla f= \left[\matrix{2\cr 2x\cr 2z}\right].\]
For $f=\vec y^T\vec x=\vec y \wedge \vec x,$,
\[\nabla f=\left[\matrix{y\cr x}\right].\]
And for $f=||\vec x||^2$,
\[\nabla f=2 \vec x.\]

We take the derivative of $E$ in equation /ref{flowMSE} with respect to 
each of its three paramters, and use some of the properties of the equations
above:
\[\frac{\partial E}{\partial \vec t} = -\frac{1}{N}\sum_{i=1}^{N}
2(\vec p_i-\vec p_i')\mu_i\hat{d}_i^2 = 0 = (3 \times 1 \: {\mbox{vector}}) \]
\[\frac{\partial E}{\partial \mu_i} = 
(\hat{d}_i^2 \vec t)^T(\vec p_i-\vec p_i') = 0 = (1 \times 1 \: {\mbox{scalar}}) \]
\[\frac{\partial E}{\partial \vec \omega} = -\frac{1}{N}\sum_{i=1}^{N}
2(\vec p_i-\vec p_i')\hat{d}_i = 0 = (3 \times 1 \: {\mbox{vector}}) \]

Useful identities:
\begin{itemize}
\item $\hat{d}_i^2\vec p_i=\vec p_i$
\item $\hat{d}_i^2\vec p_i'=\mu_i\vec t(\vec d_i^T\vec t)\vec d_i + \hat
\omega \vec d_i$
\end{itemize}

Use these identities to rewrite our error with respect to $\vec t$:
\[\frac{\partial E}{\partial \vec t} = \frac{1}{N}\sum_{i=1}^{N} \left[
-\mu_i\vec p_i - \mu_i (\mu_i\vec t - \mu_i(\vec d_i^T\vec
t)\vec d_i+\hat \omega \vec d_i\right] = 0\]
And rearrange to get:
\[\left(\frac{1}{N}\sum_{i=1}^{N} \mu_i^2\right)\vec t = -<\mu \vec p> +
<\mu^2(\vec d^T\vec t)d> - \hat \omega <\mu \vec d>\]

Note: $< \ldots >$ indicates expected value of the term inside.

To be continued in next lecture \ldots\

\end{document}

