Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Maximum Likelihood Estimation (MLE)

Mahmood Amintoosi, Fall 2026

Computer Science Dept, Ferdowsi University of Mashhad

The definitions and the figure on this page follow chapter 3 of Pattern Classification by Duda, Hart, and Stork (Duda et al. (2000)). The estimates derived here are the ones used by the Bayesian Decision Theory page, where the class-conditional densities are assumed known.

Learning objectives. By the end you should be able to

  1. write the likelihood and the log-likelihood of a Gaussian sample and say why the logarithm is taken;

  2. derive the MLE of the mean and of the variance of a univariate Gaussian;

  3. derive the MLE of the mean vector and of the covariance matrix of a multivariate Gaussian;

  4. say in what sense these estimates are biased, and when the covariance estimate is not even defined;

  5. explain how the estimates are plugged into the Bayes decision rule, and how that gives LDA and QDA.

English · فارسی

1. Introduction

In statistical pattern classification, a common assumption is that the data is generated by an underlying probability distribution with unknown parameters. For example, we may assume a feature xx follows a Gaussian (Normal) distribution. To make optimal decisions (e.g., classification using Bayes decision theory), we must estimate the parameters of these distributions from a training dataset.

Maximum Likelihood Estimation (MLE) is a standard parametric method used to estimate the unknown parameters θ\theta of a probability density function p(x∣θ)p(x|\theta) given a dataset D\mathcal{D}. The principle is to select the parameter values that make the observed data most probable.


2. The Likelihood Function

The likelihood function, denoted as L(θ∣D)L(\theta|\mathcal{D}), represents the probability of observing the specific dataset D\mathcal{D} as a function of the parameters θ\theta.

Key Distinction:
It is crucial to distinguish between the probability density function p(x∣θ)p(x|\theta) and the likelihood function L(θ∣D)L(\theta|\mathcal{D}):

  1. p(x∣θ)p(x|\theta): Describes the distribution of the data xx for fixed parameters θ\theta.

  2. L(θ∣D)L(\theta|\mathcal{D}): Describes the probability of the dataset D\mathcal{D} for fixed observed data, viewed as a function of θ\theta.

The likelihood is a function of θ\theta, defined on the parameter space; the density is a function of xx, defined on the feature space. As Duda, Hart, and Stork put it, the functional form of the likelihood is not necessarily the same as the functional form of the density.

Figure 3.1: The top graph shows several training points in one dimension, known or assumed to be drawn from a Gaussian of a particular variance, but unknown mean. Four of the infinite number of candidate source distributions are shown in dashed lines. The middle figure shows the likelihood p(D∣θ)p(\mathcal{D}|\theta) as a function of the mean. If we had a very large number of training points, this likelihood would be very narrow. The value that maximizes the likelihood is marked θ^\hat{\theta}; it also maximizes the logarithm of the likelihood — i.e., the log-likelihood ℓ(θ)\ell(\theta), shown at the bottom. Note especially that the likelihood lies in a different space from p(x∣θ^)p(x|\hat{\theta}), and the two can have different functional forms.


3. Mathematical Derivation: Univariate Gaussian

We begin with the simplest case: estimating the parameters of a univariate Gaussian distribution. This serves as the foundation for understanding the multivariate case used in pattern classification.

3.1 Problem Definition

Let D={x1,x2,…,xn}\mathcal{D} = \{x_1, x_2, \dots, x_n\} be a dataset consisting of nn independent and identically distributed (i.i.d.) samples. We assume each sample is drawn from a univariate Gaussian distribution:

p(x∣μ,σ2)=12πσ2exp⁡(−(x−μ)22σ2)p(x | \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)

where:

3.2 The Likelihood Function

The likelihood function L(μ,σ2)L(\mu, \sigma^2) is the joint probability density of the observed data, assuming independence:

L(μ,σ2)=p(D∣μ,σ2)=∏k=1np(xk∣μ,σ2)=∏k=1n12πσ2exp⁡(−(xk−μ)22σ2)=(12πσ2)nexp⁡(−∑k=1n(xk−μ)22σ2)\begin{aligned} L(\mu, \sigma^2) &= p(\mathcal{D} | \mu, \sigma^2) \\ &= \prod_{k=1}^n p(x_k | \mu, \sigma^2) \\ &= \prod_{k=1}^n \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_k - \mu)^2}{2\sigma^2}\right) \\ &= \left( \frac{1}{\sqrt{2\pi\sigma^2}} \right)^n \exp\left( -\sum_{k=1}^n \frac{(x_k - \mu)^2}{2\sigma^2} \right) \end{aligned}

3.3 The Log-Likelihood Function

Directly maximizing L(μ,σ2)L(\mu, \sigma^2) is computationally difficult due to the product of exponentials. Since the logarithm is a strictly monotonic function, maximizing the log-likelihood ℓ(μ,σ2)\ell(\mu, \sigma^2) yields the same parameter estimates as maximizing the likelihood.

ℓ(μ,σ2)=log⁡L(μ,σ2)=∑k=1nlog⁡(12πσ2exp⁡(−(xk−μ)22σ2))=∑k=1n[−12log⁡(2π)−12log⁡(σ2)−(xk−μ)22σ2]=−n2log⁡(2π)−n2log⁡(σ2)−12σ2∑k=1n(xk−μ)2\begin{aligned} \ell(\mu, \sigma^2) &= \log L(\mu, \sigma^2) \\ &= \sum_{k=1}^n \log \left( \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_k - \mu)^2}{2\sigma^2}\right) \right) \\ &= \sum_{k=1}^n \left[ -\frac{1}{2} \log(2\pi) - \frac{1}{2} \log(\sigma^2) - \frac{(x_k - \mu)^2}{2\sigma^2} \right] \\ &= -\frac{n}{2} \log(2\pi) - \frac{n}{2} \log(\sigma^2) - \frac{1}{2\sigma^2} \sum_{k=1}^n (x_k - \mu)^2 \end{aligned}

3.4 Estimating the Mean (μ\mu)

To find the Maximum Likelihood Estimate (MLE) μ^\hat{\mu}, we take the partial derivative of ℓ\ell with respect to μ\mu and set it to zero:

∂ℓ∂μ=∑k=1n(xk−μ)σ2=1σ2∑k=1n(xk−μ)=0\frac{\partial \ell}{\partial \mu} = \sum_{k=1}^n \frac{(x_k - \mu)}{\sigma^2} = \frac{1}{\sigma^2} \sum_{k=1}^n (x_k - \mu) = 0

Solving for μ\mu:

∑k=1nxk−nμ^=0  ⟹  μ^=1n∑k=1nxk\sum_{k=1}^n x_k - n\hat{\mu} = 0 \implies \hat{\mu} = \frac{1}{n} \sum_{k=1}^n x_k

Result: The MLE for the mean is the sample mean.

3.5 Estimating the Variance (σ2\sigma^2)

To find the MLE σ^2\hat{\sigma}^2, we take the partial derivative of ℓ\ell with respect to σ2\sigma^2. Let τ=σ2\tau = \sigma^2 for simpler differentiation:

∂ℓ∂τ=−n2τ+12τ2∑k=1n(xk−μ)2=0\frac{\partial \ell}{\partial \tau} = -\frac{n}{2\tau} + \frac{1}{2\tau^2} \sum_{k=1}^n (x_k - \mu)^2 = 0

Multiplying by 2τ22\tau^2:

−nτ+∑k=1n(xk−μ)2=0-n\tau + \sum_{k=1}^n (x_k - \mu)^2 = 0

Substituting μ^\hat{\mu} for μ\mu and τ=σ2\tau = \sigma^2:

σ^2=1n∑k=1n(xk−μ^)2\hat{\sigma}^2 = \frac{1}{n} \sum_{k=1}^n (x_k - \hat{\mu})^2

Result: The MLE divides by nn, not by n−1n-1. The quantity σ^2\hat{\sigma}^2 is therefore the biased version of the sample variance; the unbiased estimator is s2=1n−1∑k(xk−μ^)2s^2 = \frac{1}{n-1}\sum_k (x_k - \hat{\mu})^2 (Section 5).


4. Extension to Multivariate Gaussian

In Pattern Classification, features are typically vectors x∈Rd\mathbf{x} \in \mathbb{R}^d. The univariate derivation extends naturally to the Multivariate Gaussian Distribution.

4.1 Density Function

p(x∣μ,Σ)=1(2π)d/2∣Σ∣1/2exp⁡(−12(x−μ)TΣ−1(x−μ))p(\mathbf{x} | \boldsymbol{\mu}, \boldsymbol{\Sigma}) = \frac{1}{(2\pi)^{d/2} |\boldsymbol{\Sigma}|^{1/2}} \exp\left( -\frac{1}{2} (\mathbf{x} - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu}) \right)

4.2 MLE Estimates

The log-likelihood is the vector version of the expression in Section 3.3:

ℓ(μ,Σ)=−n2log⁡(2π)−n2log⁡∣Σ∣−12∑k=1n(xk−μ)TΣ−1(xk−μ).\ell(\boldsymbol{\mu}, \boldsymbol{\Sigma}) = -\frac{n}{2} \log(2\pi) - \frac{n}{2} \log |\boldsymbol{\Sigma}| - \frac{1}{2} \sum_{k=1}^n (\mathbf{x}_k - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x}_k - \boldsymbol{\mu}).

Because Σ\boldsymbol{\Sigma} is symmetric, and because ∂∂μaTb=a\frac{\partial}{\partial \boldsymbol{\mu}} \mathbf{a}^T \mathbf{b} = \mathbf{a}, the two derivatives are

∂ℓ∂μ=∑k=1nΣ−1(xk−μ)=0,∂ℓ∂Σ=−n2Σ−1+12Σ−1[∑k=1n(xk−μ)(xk−μ)T]Σ−1=0.\frac{\partial \ell}{\partial \boldsymbol{\mu}} = \sum_{k=1}^n \boldsymbol{\Sigma}^{-1} (\mathbf{x}_k - \boldsymbol{\mu}) = \mathbf{0}, \qquad \frac{\partial \ell}{\partial \boldsymbol{\Sigma}} = -\frac{n}{2} \boldsymbol{\Sigma}^{-1} + \frac{1}{2} \boldsymbol{\Sigma}^{-1} \left[ \sum_{k=1}^n (\mathbf{x}_k - \boldsymbol{\mu})(\mathbf{x}_k - \boldsymbol{\mu})^T \right] \boldsymbol{\Sigma}^{-1} = \mathbf{0}.

The first equation gives the mean vector, and substituting it into the second gives the covariance matrix:

  1. Mean Vector (μ\boldsymbol{\mu}):

    μ^=1n∑k=1nxk\hat{\boldsymbol{\mu}} = \frac{1}{n} \sum_{k=1}^n \mathbf{x}_k

    The MLE is the sample mean vector.

  2. Covariance Matrix (Σ\boldsymbol{\Sigma}):

    Σ^=1n∑k=1n(xk−μ^)(xk−μ^)T\hat{\boldsymbol{\Sigma}} = \frac{1}{n} \sum_{k=1}^n (\mathbf{x}_k - \hat{\boldsymbol{\mu}})(\mathbf{x}_k - \hat{\boldsymbol{\mu}})^T

    The MLE is the sample covariance matrix, again with denominator nn.

4.3 When the Estimate Does Not Exist

The sum above is a d×dd \times d matrix built from nn outer products, so rank⁡(Σ^)≤min⁡(n,d)\operatorname{rank}(\hat{\boldsymbol{\Sigma}}) \leq \min(n, d). If n<dn < d, or if two features are exactly collinear, Σ^\hat{\boldsymbol{\Sigma}} is singular and Σ^−1\hat{\boldsymbol{\Sigma}}^{-1} does not exist: the Gaussian density, the Mahalanobis distance and QDA are all undefined. There are two standard remedies:


5. Properties and Limitations

While MLE is a powerful tool, Duda, Hart, and Stork highlight several important characteristics of these estimates, particularly in the context of classification:

  1. Bias:

    • The estimate for the mean μ^\hat{\mu} is unbiased.

    • The estimate for the variance σ^2\hat{\sigma}^2 (and Σ^\hat{\boldsymbol{\Sigma}}) is biased. It systematically underestimates the true variance because it uses nn instead of n−1n-1 in the denominator.

    • Unbiased estimator: s2=1n−1∑(xk−μ^)2s^2 = \frac{1}{n-1} \sum (x_k - \hat{\mu})^2. However, for large nn, the difference is negligible.

  2. Consistency:

    • As the number of samples n→∞n \to \infty, the MLE estimates converge to the true parameter values.

  3. Invariance:

    • If θ^\hat{\theta} is the MLE of θ\theta, then for any function g(θ)g(\theta), the MLE of g(θ)g(\theta) is g(θ^)g(\hat{\theta}).

  4. Sensitivity to Outliers:

    • Because MLE assumes the data follows a specific distribution (e.g., Gaussian), outliers can significantly skew the estimates of the mean and covariance, potentially degrading classification performance.

6. Connection to Bayesian Classification

The primary role of MLE in this course is to enable Bayesian Classification when parameters are unknown.

6.1 The “Plug-In” Approach

  1. Train: For each class ωi\omega_i, use the training data Di\mathcal{D}_i to compute MLE estimates μ^i\hat{\boldsymbol{\mu}}_i and Σ^i\hat{\boldsymbol{\Sigma}}_i.

  2. Construct: Form the estimated class-conditional density: p^(x∣ωi)=N(x∣μ^i,Σ^i)\hat{p}(\mathbf{x} | \omega_i) = \mathcal{N}(\mathbf{x} | \hat{\boldsymbol{\mu}}_i, \hat{\boldsymbol{\Sigma}}_i).

  3. Decide: Apply the Bayes Decision Rule using these estimated densities.

This is known as a Plug-In Classifier. It approximates the optimal Bayes classifier by plugging in the best estimate of the parameters.

6.2 Impact on Decision Boundaries

The MLE estimates dictate the geometry of the classifier:


7. Bridge to Advanced Topics

MLE is not limited to simple Gaussian classification. It is the foundational engine for many advanced models we will cover later:


8. Summary

The Maximum Likelihood Estimation method provides a principled approach to learning parameters for probabilistic models in machine learning.

Key Takeaways


References

Duda et al. (2000)

Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern Classification (2nd ed.). Wiley-Interscience, chapter 3.

References
  1. Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern Classification (2nd ed.). Wiley-Interscience.