Bayesian Decision Theory — Core Formulas

Understanding what every symbol means is the key to mastering Bayesian decisions
Bayes Formula Core
\[P(\omega_i|x) = \frac{P(x|\omega_i) \cdot P(\omega_i)}{P(x)}\]
Meaning:The posterior probability that the sample belongs to class ωᵢ after observing feature x.
Parameters:P(x|ωᵢ) — class-conditional probability (likelihood); P(ωᵢ) — prior probability; P(x) — evidence factor (normalizing constant).
Law of Total Probability Normalization
\[P(x) = \sum_{j=1}^{c} P(x|\omega_j) \cdot P(\omega_j)\]
Meaning:The evidence factor P(x) is the total probability of producing feature x across all classes.
Parameters:c — the total number of classes; for each class j, sum the product of its class-conditional probability and its prior.
Normal Distribution Density Class-conditional
\[P(x|\omega_i) = \frac{1}{\sqrt{2\pi}\,\sigma_i} \exp\left(-\frac{(x-\mu_i)^2}{2\sigma_i^2}\right)\]
Meaning:Assumes the features of class ωᵢ follow a normal distribution, with μᵢ as the mean and σᵢ as the standard deviation.
Parameters:μᵢ — class center; σᵢ — class spread (the larger it is, the more dispersed the class).
Posterior Probability Computation Binary
\[P(\omega_i|x) = \frac{P(x|\omega_i)P(\omega_i)}{P(x|\omega_1)P(\omega_1) + P(x|\omega_2)P(\omega_2)}\]
Meaning:For a binary problem, the posterior is each class weighted likelihood as a share of the total weighted likelihood.
Parameters:The numerator is class i weighted likelihood; the denominator is the sum of the weighted likelihoods over all classes.
Minimum Error-Rate Decision Rule MAP
\[\hat{\omega} = \arg\max_{\omega_i} \; P(\omega_i|x)\]
Meaning:Pick the class with the largest posterior probability as the decision, which is equivalent to minimizing the classification error rate.
Equivalent form:if P(x|ω₁)P(ω₁) > P(x|ω₂)P(ω₂), decide ω₁.
Conditional Risk and Minimum-Risk Decision Cost-sensitive
\[R(\alpha_i|x) = \sum_{j=1}^{c} \lambda(\alpha_i|\omega_j) \cdot P(\omega_j|x)\]
Meaning:The conditional risk of decision αᵢ (deciding class i) is the loss-weighted sum over all possible true classes.
Decision rule:choose the decision with the smallest R(αᵢ|x), i.e. α* = arg min R(αᵢ|x).
Parameters:λ(αᵢ|ωⱼ) — the loss of deciding ωᵢ when the true class is ωⱼ (a correct decision usually has zero loss).

Interactive Simulation

Adjust the parameters on the left and watch the probability distributions and the decision boundary change in real time
Parameter Control
P(ω₂) = 1 − P(ω₁); the prior reflects each class share of the population



When the two are equal, the minimum-risk decision reduces to the minimum error-rate decision

Move the sample point and watch the classification result change
Class-Conditional Probability Density (prior-weighted)
Posterior Probability and Decision Boundary
Classification Result for the Current Sample

A Deeper Look at Bayesian Decision Theory

From principles to applications: a complete understanding of this classic decision framework

Standard Introduction

Bayesian decision theory is the optimal decision-making method within a probabilistic framework in statistical pattern recognition. It assumes that all relevant probability distributions are known (including the prior probabilities and the class-conditional densities), then uses Bayes formula to compute posterior probabilities and makes the decision that minimizes expected risk. Under the minimum error-rate criterion, the Bayesian decision amounts to choosing the class with the largest posterior probability (the MAP criterion), a rule that theoretically guarantees the smallest possible classification error rate. When a loss matrix is introduced, the Bayesian decision extends further into the minimum-risk decision, which shifts the decision boundary according to how costly each type of error is so that the expected total loss is minimized.

Plain-Language Introduction

Imagine you are deciding whether an email is spam. You would not guess out of thin air: you look at the tell-tale words in the message (“free”, “winner” and so on), then combine that with “the share of spam in all mail” and “the probability that spam contains these words”. Bayesian decision theory simply turns this intuition into mathematics: the prior probability is your initial belief about the class, the class-conditional probability is “how likely the feature is, given that class”, and the posterior probability is the final judgement after combining all the evidence. Bayesian decision theory tells us: choose the class with the largest posterior probability and you minimize the chance of being wrong.

What Problem Does It Solve

Bayesian decision theory solves the problem of optimal classification under uncertainty. In practice, feature observations are noisy and overlapping — the same feature value can come from different classes. Bayesian decision theory provides a rigorous mathematical framework for answering “given the observed feature x, which class is the sample most likely to belong to?” and “if different errors carry different costs, how should the decision be adjusted?” It gives the lower bound on the classification error rate from a probabilistic standpoint and serves as the theoretical benchmark for every classifier.

Why It Matters

Bayesian decision theory is a theoretical cornerstone of pattern recognition and machine learning. First, it defines what an optimal classifier even means: the Bayes classifier is optimal in the error-rate sense, and no other classifier can beat it (provided the true distribution is known). Second, it supplies a baseline against which other classifiers can be evaluated. Third, the minimum-risk framework introduces the central idea of cost-sensitive learning, which is crucial in fields such as medical diagnosis and financial risk control. Beyond that, Bayesian thinking has inspired a large family of practical algorithms: the naive Bayes classifier, Bayesian networks, Bayesian optimization and more.

Application Scenarios

  • Spam filtering:the naive Bayes classifier is the classic application, computing the posterior probability that a message is spam from word-frequency features.
  • Medical diagnosis:combines symptom features, the prior prevalence of a disease and test accuracy to compute the posterior probability that a patient is ill, supporting the doctor decision.
  • Financial risk control:estimates an applicant default probability and makes a minimum-risk decision using the costs of each kind of error (rejecting a good customer vs. lending to a bad one).
  • Speech recognition:acoustic models use the Bayesian framework to combine acoustic features with a language prior.
  • Computer vision:the probabilistic modelling foundation behind object detection and image classification.

Limitations and Drawbacks

  • Distribution assumptions:it requires accurate knowledge of the priors and class-conditional densities; in practice these can only be estimated, and estimation error degrades the decision.
  • Curse of dimensionality:as the feature dimension grows, estimating class-conditional densities becomes extremely hard and needs an exponential amount of data.
  • Feature independence assumption:naive Bayes assumes features are conditionally independent, which often fails in reality and can bias the posterior estimates.
  • Sensitivity to the prior:the choice of prior strongly affects the posterior, so an inappropriate prior can lead to wrong decisions.
  • Computational cost:exact Bayesian inference can be prohibitively expensive in complex models, so approximation is required.