\documentstyle[12pt,fullpage,psfig]{article}

\newcommand{\lecturetitle}[4]{
  \noindent
  CS 280, Computer Vision (Prof. Jitendra Malik) \hfill #3\\
  Lecture \##1 (#2) \hfill {\bf DRAFT} Notes by #4\\[-.05in]
  \mbox{}\hrulefill \mbox{}\\}

\begin{document}

% MODIFY THE FOLLOWING TO INCLUDE THE LECTURE #, TOPIC, DATE, and YOUR NAME
\lecturetitle{23}{Stereo Correspondence and Grouping / Segmentation}{Apr 19, 1999}{Zhanfeng Jia}

% INCLUDE AN OUTLINE OF THE CLASS TOPICS
\begin{itemize} 
  \item Stereo Correspondence
  \item Grouping / Segmentation
\end{itemize}

% THE NOTES WILL GO HERE

\section{Stereo Correspondence}

	Corresponding points have
\begin{itemize}
\item Similar values of brightness: result in high cross correlation score.
\item Piecewise smoothness of disparity field.
\item Uniqueness constraint: less than 1 corresponding point.
\item Ordering constraint.
\item Epipolar constraint: corresponding point lies on the epipolar lines.
\end{itemize}

\subsection{Find disparity -- simple minded}

	Simple consideration on epipolar constraint tells us that the matching points lie on
the correponding epipolar lines. That is, P locates on the left epipolar line while the matching
point P' locates on the right epipolar line. So we can search for P' along that line, using a
small window of size 5 by 5 or 7 by 7. See Figure 1.
\\
\\

\begin{figure}[htb]
\centerline{\psfig{file=sc01.eps,height=0.8in}}
\caption{Disparity is function of depth. The typical range may be plus/minus 10. For each possible
   P', compute match-score. Define corresponding point to be one with maximal match-score.
   (Implement as part of homework)}
\label{Figure 1}
\end{figure}

	The size of the matching window need to be chosen properly. Basicly, it is obvious that
large window is less sensitive to noise but introduces more problems at depth boudaries. See
Figure 2.
\\
\\

\begin{figure}[htb]
\centerline{\psfig{file=sc02.eps,height=2in}}
\caption{Size of window. The shade area exposes in the left image, but is covered in the right
   image. So the windows on the boundary of objects don't match well. The larger the size of
   window is, the less it matches.}
\label{Figure 2}
\end{figure}

	Figure 3 shows a good way to choose window near a boundary. It avoids the problem over
the bounary but depends on the decision of boundaries in advance, which is another hard-to-solve
problem.
\\

\begin{figure}[htb]
\centerline{\psfig{file=sc03.eps,height=2in}}
\caption{A suitable window choice near the boundary.}
\label{Figure 3}
\end{figure}

	Note the fact that when P' matches P, P should match P' at the same time. So the following
methods make the calculation more acurate:

	1. Use the average of the two calculated disparity.

	2. If the values are too different, claim no match.

\subsection{Find disparity -- using minimal cost path}

	Another property we may use in finding the matching points is continuity,
that is, the 'piecewise smoothness' property listed at the very beginning. With
these constraints, we introduce minimal cost path method. Refer to Figure 4.

	In Figure 4, we focus on the corresponding epipolar lines from left and
right images. The 'scan line' curves show the brightness value on the epipolar
lines. Cost values are calculated for each pair of pixels, using cost function
as following:
\[
cost(m,n) = cost \ of \ matching \ pixels \ m \ and \ n
\]
	where $m$ is from left epipolar line and $n$ is from right epipolar line.

	If the images contain 255 pixels each row, we get a 255*255 matrix in the
end, each of the element having a corresponding cost value.

	We obviously expect small cost value at matching points. Considering of
continuity of disparity, the job tures out to be finding a path with as small cost
values as possible frim the left-up corner of the matrix to the right-down corner.
This path is called {\bf minimal cost path}.
\\
\\

\begin{figure}[htb]
\centerline{\psfig{file=sc04.eps,height=2in}}
\caption{Cost path}
\label{Figure 4}
\end{figure}

	How do we find minimal cost path? The answer is using dynamic programming.
\\
\\

\begin{figure}[htb]
\centerline{\psfig{file=sc05.eps,height=2in}}
\caption{Dynamic programming -- finding minimal cost path}
\label{Figure 5}
\end{figure}

	Figure 5 shows the basic idea of dynamic programming. Here another property
listed ahead, uniqueness constraint, is used. With the continuity constraint, it means
that if the matrix element marked by dot is claimed to be part of the path, the last
element on the path must be one of the three ones marked by star. Thus if we know two
elements that are on the path, as showed in the Figure 5, the minimal cost path is
constrained in the area marked by dash-line.

\subsection{Stereo correspondence -- still open}

	There are still some unsolved problems.

	- regions without texture. Certain regions like a white wall make it impossible
to find out the disparity.

	- depth discontinuities, that is, regions of half-occlusion. We met this kind
of problem when choosing the size of matching window. See Figure 2, 3.

	Also, current continuity constraint is only one-dimensional case. Ohta and
Kanade extend dynamic programming and use inter-scanline constraint. Figure 6 shows the
idea.

\begin{figure}[htb]
\centerline{\psfig{file=sc06.eps,height=1.6in}}
\caption{Extended DP by Ohta and Kanade, using the information that if p and p' are close,
   q and q' should also be close.}
\label{Figure 6}
\end{figure}

\section{Grouping / Segmentation}

	Boundaries of image regions defined by a number of attributes.
\begin{itemize}
\item Brightness and color
\item Texture
\item Motion
\item Stereoscopic depth
\item Familiar configuration
\end{itemize}

\newpage

\begin{figure}[htb]
\centerline{\psfig{file=sc07.ps,height=2.8in}}
\caption{Example picture -- bedroom}
\label{Figure 7}
\end{figure}

	Factors that lead to grouping:
\begin{itemize}
\item Similarity of brightness, color, texture, disparity, motion, ...
\item Proximity
\item Good continuation of boundary contour
\item Closure
\item Symmetry and parallelism
\item Familiar configuration
\end{itemize}


\end{document}
