Maximum Likelihood Estimation (MLE)
Mahmood Amintoosi, Fall 2026
Computer Science Dept, Ferdowsi University of Mashhad
The definitions and the figure on this page follow chapter 3 of Pattern Classification by Duda, Hart, and Stork (Duda et al. (2000)). The estimates derived here are the ones used by the Bayesian Decision Theory page, where the class-conditional densities are assumed known.
Learning objectives. By the end you should be able to
write the likelihood and the log-likelihood of a Gaussian sample and say why the logarithm is taken;
derive the MLE of the mean and of the variance of a univariate Gaussian;
derive the MLE of the mean vector and of the covariance matrix of a multivariate Gaussian;
say in what sense these estimates are biased, and when the covariance estimate is not even defined;
explain how the estimates are plugged into the Bayes decision rule, and how that gives LDA and QDA.
1. Introduction¶
In statistical pattern classification, a common assumption is that the data is generated by an underlying probability distribution with unknown parameters. For example, we may assume a feature follows a Gaussian (Normal) distribution. To make optimal decisions (e.g., classification using Bayes decision theory), we must estimate the parameters of these distributions from a training dataset.
Maximum Likelihood Estimation (MLE) is a standard parametric method used to estimate the unknown parameters of a probability density function given a dataset . The principle is to select the parameter values that make the observed data most probable.
2. The Likelihood Function¶
The likelihood function, denoted as , represents the probability of observing the specific dataset as a function of the parameters .
Key Distinction:
It is crucial to distinguish between the probability density function and the likelihood function :
: Describes the distribution of the data for fixed parameters .
: Describes the probability of the dataset for fixed observed data, viewed as a function of .
The likelihood is a function of , defined on the parameter space; the density is a function of , defined on the feature space. As Duda, Hart, and Stork put it, the functional form of the likelihood is not necessarily the same as the functional form of the density.

Figure 3.1: The top graph shows several training points in one dimension, known or assumed to be drawn from a Gaussian of a particular variance, but unknown mean. Four of the infinite number of candidate source distributions are shown in dashed lines. The middle figure shows the likelihood as a function of the mean. If we had a very large number of training points, this likelihood would be very narrow. The value that maximizes the likelihood is marked ; it also maximizes the logarithm of the likelihood — i.e., the log-likelihood , shown at the bottom. Note especially that the likelihood lies in a different space from , and the two can have different functional forms.
3. Mathematical Derivation: Univariate Gaussian¶
We begin with the simplest case: estimating the parameters of a univariate Gaussian distribution. This serves as the foundation for understanding the multivariate case used in pattern classification.
3.1 Problem Definition¶
Let be a dataset consisting of independent and identically distributed (i.i.d.) samples. We assume each sample is drawn from a univariate Gaussian distribution:
where:
is the mean (location parameter).
is the variance (scale parameter).
3.2 The Likelihood Function¶
The likelihood function is the joint probability density of the observed data, assuming independence:
3.3 The Log-Likelihood Function¶
Directly maximizing is computationally difficult due to the product of exponentials. Since the logarithm is a strictly monotonic function, maximizing the log-likelihood yields the same parameter estimates as maximizing the likelihood.
3.4 Estimating the Mean ()¶
To find the Maximum Likelihood Estimate (MLE) , we take the partial derivative of with respect to and set it to zero:
Solving for :
Result: The MLE for the mean is the sample mean.
3.5 Estimating the Variance ()¶
To find the MLE , we take the partial derivative of with respect to . Let for simpler differentiation:
Multiplying by :
Substituting for and :
Result: The MLE divides by , not by . The quantity is therefore the biased version of the sample variance; the unbiased estimator is (Section 5).
4. Extension to Multivariate Gaussian¶
In Pattern Classification, features are typically vectors . The univariate derivation extends naturally to the Multivariate Gaussian Distribution.
4.1 Density Function¶
4.2 MLE Estimates¶
The log-likelihood is the vector version of the expression in Section 3.3:
Because is symmetric, and because , the two derivatives are
The first equation gives the mean vector, and substituting it into the second gives the covariance matrix:
Mean Vector ():
The MLE is the sample mean vector.
Covariance Matrix ():
The MLE is the sample covariance matrix, again with denominator .
4.3 When the Estimate Does Not Exist¶
The sum above is a matrix built from outer products, so . If , or if two features are exactly collinear, is singular and does not exist: the Gaussian density, the Mahalanobis distance and QDA are all undefined. There are two standard remedies:
reduce the dimension first (PCA, feature selection);
regularise the estimate, with , which is what “shrinkage” and the LDA implementation of Scikit-Learn do.
5. Properties and Limitations¶
While MLE is a powerful tool, Duda, Hart, and Stork highlight several important characteristics of these estimates, particularly in the context of classification:
Bias:
The estimate for the mean is unbiased.
The estimate for the variance (and ) is biased. It systematically underestimates the true variance because it uses instead of in the denominator.
Unbiased estimator: . However, for large , the difference is negligible.
Consistency:
As the number of samples , the MLE estimates converge to the true parameter values.
Invariance:
If is the MLE of , then for any function , the MLE of is .
Sensitivity to Outliers:
Because MLE assumes the data follows a specific distribution (e.g., Gaussian), outliers can significantly skew the estimates of the mean and covariance, potentially degrading classification performance.
6. Connection to Bayesian Classification¶
The primary role of MLE in this course is to enable Bayesian Classification when parameters are unknown.
6.1 The “Plug-In” Approach¶
Train: For each class , use the training data to compute MLE estimates and .
Construct: Form the estimated class-conditional density: .
Decide: Apply the Bayes Decision Rule using these estimated densities.
This is known as a Plug-In Classifier. It approximates the optimal Bayes classifier by plugging in the best estimate of the parameters.
6.2 Impact on Decision Boundaries¶
The MLE estimates dictate the geometry of the classifier:
Linear Discriminant Analysis (LDA): Assumes all classes share a common covariance . The decision boundary becomes linear (hyperplane).
Quadratic Discriminant Analysis (QDA): Assumes classes have unique covariances . The decision boundary becomes a quadric surface (an ellipse, hyperbola or parabola in two dimensions, a hyperquadric in general).
7. Bridge to Advanced Topics¶
MLE is not limited to simple Gaussian classification. It is the foundational engine for many advanced models we will cover later:
Principal Component Analysis (PCA): Can be derived as a Maximum Likelihood problem for a Gaussian model with constraints (finding directions of maximum variance).
Gaussian Mixture Models (GMM): Extends single Gaussians to mixtures. MLE is used, but requires the EM Algorithm (Expectation-Maximization) for iterative solution.
Hidden Markov Models (HMM): Parameters (transition and emission probabilities) are estimated by MLE with the Baum–Welch algorithm, an EM variant that uses the forward–backward recursions to compute the expected counts.
8. Summary¶
The Maximum Likelihood Estimation method provides a principled approach to learning parameters for probabilistic models in machine learning.
Key Takeaways¶
Likelihood vs. Probability: We maximize the likelihood of the data given the parameters, not the probability of the parameters given the data.
Log-Likelihood: Transforming to the log-domain converts products to sums, simplifying optimization.
Univariate Results:
, the sample mean (unbiased);
, the biased estimate of the variance (the unbiased sample variance divides by ).
Relevance to Classification: These estimates are the building blocks for Linear Discriminant Analysis (LDA) and other Gaussian-based classifiers discussed later in the course.
References¶
Duda et al. (2000)
Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern Classification (2nd ed.). Wiley-Interscience, chapter 3.
- Duda, R. O., Hart, P. E., & Stork, D. G. (2000). Pattern Classification (2nd ed.). Wiley-Interscience.