Maximum Likelihood Estimation (MLE) Interactive Simulation

By generating sample data and observing changes in the likelihood function, intuitively understand the core statistical idea of "the parameter value most likely to produce the observed data."

Likelihood Function

\[ L(\theta \mid \mathbf{x}) = P(\mathbf{x} \mid \theta) = \prod_{i=1}^{n} f(x_i; \theta) \]

The likelihood function L(θ|x) represents the probability (or probability density) of observing the sample data x = (x₁, x₂, ..., xₙ) given the parameter θ. The difference from probability is: probability fixes the parameter and varies the data; likelihood fixes the data and varies the parameter. The larger the likelihood value, the more "likely" that parameter value is to produce these data.

Parameters: θ — the unknown parameter to be estimated (can be a scalar or a vector); x — the observed sample; n — the sample size; f(xᵢ;θ) — the probability density function or probability mass function of a single observation.

Log-Likelihood Function

\[ \ell(\theta) = \log L(\theta \mid \mathbf{x}) = \sum_{i=1}^{n} \log f(x_i; \theta) \]

Since the likelihood function is the product of many small probability values, it is very prone to numerical underflow. Taking the logarithm converts multiplication into addition, making computation stable and optimization convenient. The logarithmic function is monotonically increasing, so maximizing the log-likelihood is equivalent to maximizing the likelihood itself.

Parameters: ℓ(θ) — the log-likelihood function; log — usually the natural logarithm (ln); the remaining symbols are consistent with the definition of the likelihood function. The logarithmic transformation does not change the location of the extreme point.

Definition of the MLE Estimator

\[ \hat{\theta}_{\text{MLE}} = \arg\max_{\theta} \, L(\theta) = \arg\max_{\theta} \, \ell(\theta) \]

The maximum likelihood estimator θ̂MLE is the parameter value that maximizes the likelihood function (or equivalently the log-likelihood function). The solution method is usually to differentiate the log-likelihood function and set it to zero, that is, to solve ∂ℓ/∂θ = 0.

Parameters: θ̂MLE — the maximum likelihood estimate; argmax — take the parameter that maximizes the function; θ ranges over all possible values in the parameter space Θ.

Analytical Solution of MLE for the Normal Distribution

\[ \hat{\mu} = \frac{1}{n}\sum_{i=1}^{n} x_i, \quad \hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^{n}(x_i - \hat{\mu})^2 \]

For the normal distribution N(μ, σ²), the MLE of μ is the sample mean (unbiased), and the MLE of σ² is the sample variance (biased, with denominator n rather than n−1). When n is large, the bias is negligible.

Parameters: μ — the mean parameter of the normal distribution; σ² — the variance parameter; xᵢ — the i-th observation; n — the sample size. μ̂ is the sample mean, and σ̂² is the second central sample moment.

Analytical Solution of MLE for the Bernoulli Distribution

\[ \hat{p} = \frac{1}{n}\sum_{i=1}^{n} x_i = \frac{k}{n} \]

For the Bernoulli distribution Bernoulli(p), the MLE of the parameter p is simply the proportion of "successes" (xᵢ=1) in the sample. This is the most intuitive estimate—using frequency to estimate probability.

Parameters: p — the probability of success in a single trial (0 ≤ p ≤ 1); k — the number of successes in the sample; n — the total number of trials; xᵢ ∈ {0,1}.

Likelihood Function Expression for the Normal Distribution

\[ L(\mu, \sigma^2) = (2\pi\sigma^2)^{-n/2} \exp\!\left(-\frac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i - \mu)^2\right) \]

Obtained by substituting the normal density function into the definition of the likelihood function. Maximizing this function allows us to take partial derivatives with respect to μ and σ² separately. When σ² is fixed, L as a function of μ has a Gaussian shape, with its peak at the sample mean.

Parameters: π — pi; exp — the exponential function; σ² — the variance (appearing as a parameter in the likelihood function); Σ(xᵢ−μ)² — the sum of squared deviations, measuring how far the data deviate from the mean.

Interactive Simulation Experiment

Adjust parameters → generate samples → observe changes in the likelihood function and the MLE estimation result

Current value: 1.80
Current value: 1.50
Current value: 60

MLE Estimation Results

Waiting to generate...

Sample Data and Distribution Fit

Likelihood Function L(μ)

Likelihood Function L(σ)

In-Depth Understanding of Maximum Likelihood Estimation

Standard Introduction

Maximum Likelihood Estimation (MLE) is one of the most important and widely used parameter estimation methods in statistics, systematically developed by the statistician R. A. Fisher in the early 20th century. Its core idea can be summarized as: among all possible parameter values, choose the one that maximizes the probability (or probability density) of observing the current sample data as the estimate.

Formally, let the probability density (or probability mass) function of the population distribution be f(x; θ), where θ is the unknown parameter. For independent and identically distributed samples x = (x₁, ..., xₙ), the likelihood function is defined as the joint probability density:

\[ L(\theta) = \prod_{i=1}^{n} f(x_i; \theta) \]

The maximum likelihood estimator θ̂MLE is the parameter value that maximizes L(θ) (or equivalently maximizes the log-likelihood ℓ(θ) = log L(θ)). Due to the monotonicity of the logarithmic function, the solutions of the two optimizations are the same, while the logarithmic transformation converts products into sums, which is more convenient computationally.

Plain-Language Introduction

Imagine you walk into a restaurant and find that there are especially many customers today, and you want to know the average waiting time at this restaurant. You randomly ask 10 customers and record their waiting times: 5 minutes, 8 minutes, 12 minutes, 6 minutes, 9 minutes... So what is the "true average waiting time" of this restaurant most likely to be?

Intuition tells us: take the average of these 10 numbers. But why is the average the most reasonable? Maximum likelihood estimation provides a rigorous mathematical explanation: if we assume that the waiting time follows a normal distribution, then the mean parameter that maximizes the probability of "observing exactly these 10 data points" is precisely the arithmetic mean of these 10 numbers.

In other words, MLE is asking: "Under what parameter setting would the data I observed be the least surprising?" Those parameter values that make the data appear "very natural" are good estimates.

What Problem Does It Solve

Maximum likelihood estimation solves the core problem in statistical inference: how to make reasonable estimates of the unknown parameters of a population starting from limited sample data. Specifically, this includes:

  • A unified framework for parameter estimation: Regardless of the distributional form, MLE provides a systematic method for constructing estimators; one only needs to write down the likelihood function and then maximize it.
  • Replacing intuitive estimates: Many "obvious" estimates (such as using the mean to estimate the expectation, or using frequency to estimate probability) can be rigorously derived through MLE, thereby gaining a theoretical basis.
  • Providing a measure of estimation precision: The large-sample properties of MLE (asymptotic normality) allow us to construct confidence intervals and perform hypothesis testing.

Why It Matters

Maximum likelihood estimation occupies a central position in statistics and Machine Learning for reasons including:

  • Asymptotic optimality: Under large-sample conditions, MLE is asymptotically unbiased, consistent, and asymptotically efficient (reaching the Cramér-Rao lower bound), meaning that no other estimator can be significantly better than MLE in large samples.
  • Invariance: If θ̂ is the MLE of θ, then for any function g(θ), g(θ̂) is also the MLE of g(θ). This property keeps MLE consistent under parameter transformations.
  • Foundation of Machine Learning: The training processes of many algorithms, such as logistic regression, naive Bayes, Gaussian mixture models, and hidden Markov models, are essentially maximum likelihood estimation (or its variants).
  • Deep connection with information theory: Maximizing the likelihood is equivalent to minimizing the KL divergence, that is, finding the model distribution "closest" to the true distribution.

Application Scenarios

  • Machine Learning classification: Logistic regression learns the classification boundary by maximizing the likelihood.
  • Natural language processing: Language models (such as n-gram models) use MLE to estimate the probability of word sequences.
  • Biostatistics: Estimating the distribution parameters of drug efficacy and gene expression levels.
  • Financial engineering: Estimating the volatility parameters of asset returns (such as in GARCH models).
  • Signal processing: Estimating noise parameters and parameter estimation in signal detection.
  • Recommendation systems: Estimating latent factor parameters in matrix factorization models.

Limitations and Drawbacks

  • Small-sample bias: MLE may produce bias when the sample size is small. For example, the MLE of the variance of a normal distribution is biased (too small), and requires an n−1 correction.
  • Sensitive to model assumptions: If the true distribution does not match the assumed distribution family, MLE may give misleading results. When the model is misspecified, the estimate may be far from the true parameter.
  • Risk of overfitting: When model complexity is high, there are many parameters, and samples are few, MLE may overfit the training data, leading to degraded generalization performance. This is also the reason why regularization methods (such as MAP and Bayesian estimation) were introduced.
  • Computational challenges: For complex models, the likelihood function may be non-convex and multimodal, and numerical optimization may get stuck in a local optimum rather than the global optimum.
  • Lack of uncertainty quantification: MLE only gives a point estimate and does not directly provide information about the uncertainty of the parameter. Methods such as the Fisher information matrix or bootstrap are needed to obtain confidence intervals.