Bayesian Decision Theory — Core Formulas
ωᵢ after observing feature x.Parameters:
P(x|ωᵢ) — class-conditional probability (likelihood); P(ωᵢ) — prior probability; P(x) — evidence factor (normalizing constant).
P(x) is the total probability of producing feature x across all classes.Parameters:
c — the total number of classes; for each class j, sum the product of its class-conditional probability and its prior.
ωᵢ follow a normal distribution, with μᵢ as the mean and σᵢ as the standard deviation.Parameters:
μᵢ — class center; σᵢ — class spread (the larger it is, the more dispersed the class).
Parameters:The numerator is class
i weighted likelihood; the denominator is the sum of the weighted likelihoods over all classes.
Equivalent form:if
P(x|ω₁)P(ω₁) > P(x|ω₂)P(ω₂), decide ω₁.
αᵢ (deciding class i) is the loss-weighted sum over all possible true classes.Decision rule:choose the decision with the smallest
R(αᵢ|x), i.e. α* = arg min R(αᵢ|x).Parameters:
λ(αᵢ|ωⱼ) — the loss of deciding ωᵢ when the true class is ωⱼ (a correct decision usually has zero loss).
Interactive Simulation
A Deeper Look at Bayesian Decision Theory
Standard Introduction
Bayesian decision theory is the optimal decision-making method within a probabilistic framework in statistical pattern recognition. It assumes that all relevant probability distributions are known (including the prior probabilities and the class-conditional densities), then uses Bayes formula to compute posterior probabilities and makes the decision that minimizes expected risk. Under the minimum error-rate criterion, the Bayesian decision amounts to choosing the class with the largest posterior probability (the MAP criterion), a rule that theoretically guarantees the smallest possible classification error rate. When a loss matrix is introduced, the Bayesian decision extends further into the minimum-risk decision, which shifts the decision boundary according to how costly each type of error is so that the expected total loss is minimized.
Plain-Language Introduction
Imagine you are deciding whether an email is spam. You would not guess out of thin air: you look at the tell-tale words in the message (“free”, “winner” and so on), then combine that with “the share of spam in all mail” and “the probability that spam contains these words”. Bayesian decision theory simply turns this intuition into mathematics: the prior probability is your initial belief about the class, the class-conditional probability is “how likely the feature is, given that class”, and the posterior probability is the final judgement after combining all the evidence. Bayesian decision theory tells us: choose the class with the largest posterior probability and you minimize the chance of being wrong.
What Problem Does It Solve
Bayesian decision theory solves the problem of optimal classification under uncertainty. In practice, feature observations are noisy and overlapping — the same feature value can come from different classes. Bayesian decision theory provides a rigorous mathematical framework for answering “given the observed feature x, which class is the sample most likely to belong to?” and “if different errors carry different costs, how should the decision be adjusted?” It gives the lower bound on the classification error rate from a probabilistic standpoint and serves as the theoretical benchmark for every classifier.
Why It Matters
Bayesian decision theory is a theoretical cornerstone of pattern recognition and machine learning. First, it defines what an optimal classifier even means: the Bayes classifier is optimal in the error-rate sense, and no other classifier can beat it (provided the true distribution is known). Second, it supplies a baseline against which other classifiers can be evaluated. Third, the minimum-risk framework introduces the central idea of cost-sensitive learning, which is crucial in fields such as medical diagnosis and financial risk control. Beyond that, Bayesian thinking has inspired a large family of practical algorithms: the naive Bayes classifier, Bayesian networks, Bayesian optimization and more.
Application Scenarios
- Spam filtering:the naive Bayes classifier is the classic application, computing the posterior probability that a message is spam from word-frequency features.
- Medical diagnosis:combines symptom features, the prior prevalence of a disease and test accuracy to compute the posterior probability that a patient is ill, supporting the doctor decision.
- Financial risk control:estimates an applicant default probability and makes a minimum-risk decision using the costs of each kind of error (rejecting a good customer vs. lending to a bad one).
- Speech recognition:acoustic models use the Bayesian framework to combine acoustic features with a language prior.
- Computer vision:the probabilistic modelling foundation behind object detection and image classification.
Limitations and Drawbacks
- Distribution assumptions:it requires accurate knowledge of the priors and class-conditional densities; in practice these can only be estimated, and estimation error degrades the decision.
- Curse of dimensionality:as the feature dimension grows, estimating class-conditional densities becomes extremely hard and needs an exponential amount of data.
- Feature independence assumption:naive Bayes assumes features are conditionally independent, which often fails in reality and can bias the posterior estimates.
- Sensitivity to the prior:the choice of prior strongly affects the posterior, so an inappropriate prior can lead to wrong decisions.
- Computational cost:exact Bayesian inference can be prohibitively expensive in complex models, so approximation is required.