Naive Bayes Classifier — Core Formulas

The following formulas form the theoretical foundation of the Naive Bayes classifier. Each formula is accompanied by its parameters.

① Bayes' Theorem
$$P(Y|X) = \frac{P(X|Y) \cdot P(Y)}{P(X)}$$
Bayes' theorem describes the posterior probability of class Y given the observed data X. It combines prior knowledge with observed evidence and is the cornerstone of Bayesian statistical inference.
Parameters:
• P(Y|X) — posterior probability: the probability that the class is Y given features X
• P(X|Y) — likelihood: the probability of observing features X under class Y
• P(Y) — prior probability: the proportion of class Y in the population
• P(X) — evidence factor: the marginal probability of features X (normalization constant)
② Conditional Independence Assumption (Naive Assumption)
$$P(X|Y) = \prod_{i=1}^{n} P(x_i | Y)$$
The "naive" in Naive Bayes comes from this assumption: given class Y, the features x₁, x₂, …, xₙ are mutually independent. This assumption greatly simplifies the computational complexity, allowing the joint likelihood to be decomposed into the product of individual feature likelihoods.
Parameters:
• n — total number of features
• xᵢ — value of the i-th feature
• P(xᵢ|Y) — conditional probability of the i-th feature under class Y
③ Naive Bayes Classifier Decision Rule
$$\hat{y} = \arg\max_{y \in \mathcal{Y}} \; P(y) \prod_{i=1}^{n} P(x_i | y)$$
The classifier selects the class that maximizes the posterior probability as the prediction result. Since the evidence factor P(X) is the same for all classes, it can be omitted. In practice, the logarithmic form is often used to avoid numerical underflow.
Parameters:
• ŷ — the predicted class label
• 𝒴 — the set of all possible classes
• arg max — take the parameter value that maximizes the expression
④ Gaussian Naive Bayes Likelihood Function
$$P(x_i | y) = \frac{1}{\sqrt{2\pi\sigma_{y,i}^2}} \exp\left(-\frac{(x_i - \mu_{y,i})^2}{2\sigma_{y,i}^2}\right)$$
When features are continuous, it is assumed that each feature follows a Gaussian (normal) distribution under each class. By estimating the mean and variance of each feature under each class from the training data, the likelihood probability of a new sample can be computed.
Parameters:
• μ_{y,i} — mean of feature i under class y
• σ_{y,i} — standard deviation of feature i under class y
• xᵢ — the i-th feature value of the sample to be classified
⑤ Laplace Smoothing (Discrete Features)
$$P(x_i | y) = \frac{N_{y,x_i} + \alpha}{N_y + \alpha \cdot K}$$
When handling discrete features, some feature values may not appear in the training set, resulting in zero probability. Laplace smoothing avoids the zero-probability problem by adding a pseudo-count α, ensuring that all feature values have non-zero probability.
Parameters:
• N_{y,xᵢ} — number of samples in class y with feature value xᵢ
• N_y — total number of samples in class y
• α — smoothing parameter (usually α=1)
• K — number of possible values of that feature
⑥ Prior Probability Estimation
$$P(y) = \frac{N_y}{N}$$
The prior probability reflects the proportion of each class in the population and is estimated from the sample frequency of each class in the training set. The prior probability has an important influence on the posterior probability, especially in scenarios with imbalanced samples.
Parameters:
• N_y — number of samples in class y
• N — total number of samples in the training set
⑦ Log-Space Computation (Numerical Stability)
$$\log P(y|X) = \log P(y) + \sum_{i=1}^{n} \log P(x_i | y)$$
When the number of features is large, the product of many small probabilities leads to numerical underflow. By using a logarithmic transformation, multiplication is converted into addition, effectively improving numerical stability; this is standard practice in engineering.
Parameters:
• log — usually the natural logarithm (ln)
• The maximum a posteriori estimate in log space is equivalent to that in the original space
• The probability can finally be recovered via softmax or normalization

Interactive Simulation — Gaussian Naive Bayes Classifier

Adjust the distribution parameters of the two classes and the position of the test point, and observe the classification decision process in real time

Class A Parameters
-3.00
-2.00
1.80
1.80
Class B Parameters
3.00
2.00
1.50
1.50
Test Sample
0.00
0.00
Prior Probabilities
0.50

P(B) = 1 - P(A) = 0.50

Decision Regions and Data Distribution Two-dimensional feature space
Posterior Probability Comparison P(A|X) vs P(B|X)
Predicted class: —
P(A|X) = —
P(B|X) = —

In-Depth Understanding of the Naive Bayes Classifier

From basic principles to engineering practice, a comprehensive mastery of this classic Machine Learning algorithm

📘 Standard Introduction
The Naive Bayes Classifier is a family of probabilistic classification algorithms based on Bayes' theorem and the conditional independence assumption. It estimates the prior probability of each class and the conditional probability distribution of each feature under each class from the training data, and then, for a new sample, computes its posterior probability of belonging to each class and selects the class with the largest posterior probability as the prediction result. Depending on the feature type, Naive Bayes has several variants: Gaussian Naive Bayes (continuous features), Multinomial Naive Bayes (count features), and Bernoulli Naive Bayes (binary features). Although the "naive" assumption often does not hold in reality, this algorithm performs excellently in tasks such as text classification and spam filtering, and its simplicity and efficiency make it one of the most widely used baseline algorithms in the Machine Learning field.
💡 Plain-Language Introduction
Imagine you are determining whether an email is spam. You see keywords such as "free," "winner," and "click the link" in the email. The Naive Bayes approach is: separately count how frequently these words appear in spam, and how frequently they appear in normal emails. When a new email arrives, combining the occurrences of these words, it computes "the probability that this email is spam" and "the probability that it is a normal email," and whichever probability is larger determines the class. The so-called "naive" refers to its assumption that each word does not affect the others— although in reality "free" and "winner" often appear together, even ignoring this correlation, the classification performance is still very good. By analogy to a doctor's diagnosis: based on the patient's various examination indicators (temperature, blood pressure, lab results, etc.), separately count the distribution patterns of each indicator in the "sick" and "healthy" populations, and then comprehensively determine which class the new patient most likely belongs to.
🎯 What Problem Does It Solve
The Naive Bayes classifier solves supervised classification problems: given labeled training data, learn a mapping function from the feature space to class labels. Specifically, this includes:
  • Text classification: automatically categorizing documents into predefined categories (news classification, topic identification, etc.)
  • Spam detection: distinguishing normal emails from spam
  • Sentiment analysis: determining whether user reviews are positive, negative, or neutral
  • Medical diagnosis assistance: predicting disease likelihood based on symptoms and examination indicators
  • Real-time prediction scenarios: applications that require fast classification decisions
Its core advantages are: fast training (only one pass over the data is needed to collect statistics), fast prediction (only products need to be computed), small memory footprint (only statistics are stored), effective on small-scale data, and capable of handling high-dimensional features.
⭐ Why It Matters
The Naive Bayes classifier has milestone significance in the Machine Learning field:
  • Concise and elegant theory: it applies Bayesian statistical theory directly to classification problems and is one of the most basic instances of probabilistic graphical models
  • Strong engineering practicality: in real scenarios such as spam filtering and text classification, its accuracy is often comparable to or even better than complex models (such as SVM and random forest)
  • Natural probabilistic output: it not only gives the classification result but also provides the posterior probability of each class, making it suitable for scenarios that require confidence assessment
  • Incremental learning friendly: when new data arrives, only the statistics need to be updated, without retraining the entire model
  • Status as a baseline algorithm: almost all classification task experiments use Naive Bayes as a baseline for comparison
  • High teaching value: it is the best introductory algorithm for understanding probabilistic models, Bayesian inference, and generative classifiers
🔧 Application Scenarios
The Naive Bayes classifier has a wide range of applications in practical engineering:
  • Email systems: email providers such as Gmail and Outlook use Bayesian filters to identify spam, with accuracy reaching over 99%
  • News aggregation platforms: automatically classifying news articles into sections such as "technology," "sports," and "finance"
  • E-commerce recommendation systems: predicting purchase intent based on users' historical behavior for personalized recommendations
  • Social media analysis: automatically identifying the sentiment orientation of comments and monitoring brand reputation
  • Medical information systems: assisting doctors in preliminary diagnosis and risk stratification based on symptom combinations
  • Network security: detecting anomalies in network traffic and identifying potential intrusion behavior
  • Document management systems: automatically tagging, archiving, and retrieving documents
  • Real-time customer service systems: automatically routing user inquiries to the most appropriate customer service department
⚠️ Limitations and Drawbacks
Although the Naive Bayes classifier has many advantages, its limitations should not be overlooked:
  • The conditional independence assumption is overly idealized: in reality, features are often correlated (such as "height" and "weight"), and ignoring these associations can cause probability estimates to deviate from the true distribution. When features are strongly correlated, the estimation of the posterior probability may be excessively amplified or reduced
  • Sensitive to data distribution: Gaussian Naive Bayes assumes that features follow a normal distribution; if the actual distribution is skewed, multimodal, or heavy-tailed, classification performance will drop significantly
  • Zero-probability problem: for discrete features, if a certain feature value does not appear in the training set, the likelihood probability is zero, causing the entire posterior probability to be zero (this can be mitigated by Laplace smoothing, but the choice of the smoothing parameter needs tuning)
  • Not suitable for complex decision boundaries: Naive Bayes is a linear classifier (when the variances are equal), and cannot learn complex nonlinear decision boundaries (although different variances can produce quadratic boundaries, it is still limited)
  • Probability calibration bias: due to the violation of the conditional independence assumption, the output posterior probabilities are often overly extreme (close to 0 or 1), and the credibility of the probability values is not as good as that of discriminative models such as logistic regression
  • Sensitive to missing data: if a certain feature is missing, special handling is required; otherwise, the likelihood of that feature cannot be computed
  • Not suitable for tasks with complex interactions among features: in fields such as image recognition and speech recognition, methods such as deep learning can better capture the complex relationships among features