CS 294-2, Grouping and Recognition (Prof. Jitendra Malik) Aug 30, 1999 Lecture #3 (Maximum likelihood framework) DRAFT Notes by Joshua Levy

• Maximum likelihood framework

– The estimation problem

– Maximum likelihood method for the Gaussian

– Maximum likelihood method for biased coin and die

• Mixture models

– Motivation: piecewise-constant segmentation problem

– Incomplete data

– Dependence between variables

– Maximum likelihood method with complete data

In this lecture we begin a summary of some statistical techniques that will be useful in visual grouping problems. Generally, these fall into a few groups:

1. Maximum likelihood framework (“classical” or “objective” method)

2. Bayesian framework (“subjective” method)

3. Spectral graph theory

Today we focus on the first framework. Then, motivated by a segmentation problem, we discuss mixture models and begin to apply the framework to them.

1 Maximum likelihood framework

1.1 The estimation problem

In the traditional estimation problem in statistics, we have a random variable X drawn from some population. (For example, X could represent the height of a randomly chosen Californian.) The population has some distribution, which is often assumed to have a special form. Everyone’s favorite continuous distribution is the Gaussian, or normal distribution, which is expressed by^{1}

f(x) = \frac{1}{\sqrt{2π}σ} e^{− \frac{1}{2}(\frac{x−μ}{σ})^{2}},

and abbreviated N(μ, σ^{2}), where μ, the mean, and σ^{2}, the variance, are the parameters of the distribution. Suppose we take a sample of N values of the random variable, x_{1}, …, x_{N} (or more shortly (x_{i})), chosen independently from one another (one says the values are “independent, identically distributed” or “i.i.d.”). From the sample, we wish to estimate something about the distribution, such as μ or σ^{2} in the Gaussian.

1What this means is that the chance that X is between x and x + Δx approaches f(x)Δx as Δx approaches 0.

It’s not clear a priori what is the best way to estimate a given parameter. For example, one might guess that the average of the x_{i} estimates the mean μ, but is this estimate better than the median? Or the average of the minimum and the maximum of the x_{i}?

1.2 Maximum likelihood method for the Gaussian

The maximum likelihood framework provides a way of choosing an estimator that is quite good. Specifically, it is the best estimation asymptotically as the sample size N becomes large. (Other estimators can be better for small N, but we will content ourselves with this one.) For a proof of this property, see a statistics text; here we simply show the method of determining the estimator.

For concreteness, we will assume our population is Gaussian, though the framework can be used for any distribution. If we have a sample (x_{i}) of size N, then the likelihood of observing a particular x_{i} is

P (x_{i}) = \frac{1}{\sqrt{2π}σ} e^{− \frac{1}{2}(\frac{x_{i}−μ}{σ})^{2}}.

The likelihood of seeing all the data is ∏_{i=1}^{N} P (x_{i}). Viewing this as a function of the parameters, we define

L (μ, σ; (x_{i})) = ∏_{i=1}^{N} \frac{1}{\sqrt{2π}σ} e^{− \frac{1}{2}(\frac{x_{i}−μ}{σ})^{2}}.

Intuitively, this is how likely it is the population has parameters μ and σ given the sample (x_{i}). The maximum likelihood estimators for the parameters are the values μ̂ and σ̂ that maximize L (μ, σ; (x_{i}))

It is convenient (and equivalent) to maximize the “log-likelihood” l = ln L (μ, σ; (x_{i})). We have

l = ∑_{i=1}^{N} [ln \frac{1}{\sqrt{2π}} − ln σ − \frac{1}{2} (\frac{x_{i} − μ}{σ})^{2}] .

Note

\frac{∂l}{∂μ} = \frac{1}{σ^{2}} ∑_{i=1}^{N}(x_{i} − μ),

so if ∂l/∂μ = 0, then μ = ∑ x_{i}/N. It’s not hard to check this is indeed a minimum for l, so the maximum likelihood estimator for the mean of a Gaussian is μ̂ = ∑ x_{i}/N, the average of the sample. This is not surprising, but it does mean no other estimate (such as the median) can do better asymptotically.

With some further computation, we can see that the maximum likelihood estimator for the variance is^{2} σ̂^{2} = ∑_{i=1}^{N} ∑(x_{i} − μ̂)^{2}/N.

2Those acquainted with the usual definition of sample variance may wonder why N is not replaced by N − 1 in this expression. The discrepancy arises because the maximum likelihood estimator is best only asymptotically. The estimate we get here is in fact biased, especially for small n.

1.3 Maximum likelihood method for biased coin and die

Suppose we flip a coin that turns up heads with probability θ. We’ll now find the maximum likelihood estimator for θ. Take a sample of N flips (x_{i}). Say we get n_{1} heads. Then the likelihood is L(θ; (x_{i})) = θ^{n_{1}}(1−θ)^{N−n_{1}} and the log-likelihood is l = n_{1} ln θ+(N−n_{1}) ln(1−θ). One can check that l is minimized when

\frac{∂l}{∂θ} = \frac{n_{1}}{θ} − \frac{N − n_{1}}{1 − θ} = 0,

or θ = n_{1}/N. Again, this is an intuitively appealing answer.

Similarly, we can estimate the probabilities π_{1}, …, π_{6} of a biased die rolling numbers 1, …, 6. We observe n_{1} ones, n_{2} twos, etc. If we do a similar computation (remembering to maximize likelihood subject to the constraint ∑ π_{i} = 1), we get π̂_{k} = n_{k}/ ∑ n_{k}.

2 Mixture models

2.1 Motivation: piecewise-constant segmentation problem

Suppose we have an image of pixels, each characterized by a single number indicating its brightness. (For simplicity, we ignore color at the moment.) The image consists of various regions representing objects that are piecewise constant in brightness. However, the image will have “noise,” so each region will have some fluctuations. We wish to identify these regions. This is the piecewise-constant segmentation problem.

2.2 Incomplete data

We assume that in region k of the image, the pixel brightness is N(μ_{k}, σ_{k}^{2}). Now if we already knew the exact position of the regions, we could use the method we’ve just described to estimate brightnesses in each region. Of course, in the segmentation problem, we don’t have this information. We can think of each pixel as having two pieces of information: its brightness x_{i} and a label q_{i} saying to which region it belongs. (For now, we will assume we know there are K regions.) We know the first piece of information, but not the second. That is, we are solving an estimation problem with incomplete data.

As another example, we could take a set of observations (x_{i}, q_{i}) for N people, where each x_{i} and q_{i} represent the height and sex of a person, respectively. If we had only height measurements, how could we determine the sex of the people measured? One process for solving such problems is called expectation maximization (EM), which will be covered in the next lecture.

2.3 Dependence between variables

In the height/sex example, we have a joint probability distribution P (x, q) with two variables. We may wish to model this distribution by considering the distribution among males and among females separately. That is, we write P (x, q) = P (q)P (x|q) (where P (q) is the distribution of q alone and P (x|q) is the distribution of x given the specific choice of q) and represent this graphically:

[figure: x; q]

More generally, we can have a distribution P (x, q) of two variables with q taking values 1, 2, …, K with probability π_{1}, …, π_{K}, and P (x|q = k) = f_{k}(x; ϕ_{k}), where f_{k} is the distribution of x given q = k, and ϕ_{k} are its parameters. This is called a mixture model.

Incidentally, note that the graphical representation above can be extended to more variables. Thus

[figure: w; x; y; z]

indicates that we can write P (w, x, y, z) = P (w)P (x|w)P (y|x)P (z|x, y).

2.4 Maximum likelihood method with complete data

Say we wish to estimate the parameters π_{k} and ϕ_{k} in a mixture model given a sample of observations (x_{i}, q_{i}). As we have seen, our interest is in a sample with incomplete data (missing q_{i} in the segmentation problem), but the complete data situation is still useful to develop notation and familiarity with the problem.

We’ll find it convenient to use a (perhaps strange) notational convention: regard the value of q as a vector with K components, and represent the value k by putting a 1 in the kth component and 0’s in all other components. Thus the column vector (0, 0, 1, 0 …, 0)^{t} represents 3, and q^{k}, the kth component of q, is 0 unless q represents k.

Now the distribution function may be written

P (x, q; π_{k}, ϕ_{k}) = ∏_{k=1}^{K} [π_{k}f_{k}(x; ϕ_{k})]^{q^{k}}

so the log-likelihood function is

l = ln L(π_{k}, ϕ_{k}; x_{i}, q_{i})
= ln ∏_{i=1}^{N} ∏_{k=1}^{K} [π_{k}f_{k}(x_{i}; ϕ_{k})]^{q_{i}^{k}}

= ∑_{i=1}^{N} ∑_{k=1}^{K} q_{i}^{k} ln π_{k} + (terms not involving π_{k})

= ∑_{k=1}^{K} n_{k} ln π_{k} + (terms not involving π_{k})

where n_{k} is the number of instances of q_{i}’s taking value k. Maximizing this with respect to the π_{k} and subject to the restraint ∑ π_{k} = 1 we see the π_{k} must be chosen in the same proportions as the n_{k}. (For example, substitute π_{2} = 1 − π_{1} − π_{3} − ⋯ − π_{K} into the last expression, and note ∂l/∂π_{1} = 0 implies n_{1}/π_{1} − n_{2}/π_{2} = 0. This can be done for all pairs of π_{i}.) Hence the maximum likelihood estimators are π̂_{k} = n_{k}/ ∑ n_{k}, which is what we would expect. One could perform a similar procedure to maximize the parameters ϕ_{k} (given a choice of distributions f_{k}).