Support Vector Machine — Core Formulas
The following formulas form the theoretical foundation of SVM. Each formula is accompanied by detailed parameter and meaning explanations.
① Linear SVM Decision Function
w — weight vector, determining the normal direction of the decision boundary
x — feature vector of the input sample
b — bias term (intercept), determining the position offset of the decision boundary
② Hard Margin SVM Primal Optimization Problem
‖w‖² — squared L2 norm of the weight vector, inversely proportional to the margin width
yᵢ — label of the i-th sample, taking values +1 or -1
xᵢ — the i-th training sample
③ Soft Margin SVM Optimization Problem
C — penalty coefficient, balancing margin maximization and tolerance of misclassification
ξᵢ — slack variable, allowing the i-th sample to violate the margin constraint to some extent
n — total number of training samples
④ Lagrangian Function
αᵢ — Lagrange multiplier, corresponding to the constraint of each training sample
αᵢ ≥ 0 — non-negativity constraint ensuring the validity of the optimal solution
⑤ Dual Optimization Problem
xᵢᵀxⱼ — inner product between samples, which is the basis of the kernel function
αᵢ — dual variable, with upper bound C (consistent with the soft margin penalty coefficient)
⑥ Kernel Function Definition
K(xᵢ, xⱼ) — kernel function, computing the inner product of two samples in feature space
φ(·) — mapping function from the original space to a high-dimensional feature space
⑦ Gaussian Radial Basis (RBF) Kernel Function
γ (gamma) — kernel width parameter, controlling the influence range of a single sample
‖xᵢ - xⱼ‖² — squared Euclidean distance between two samples
⑧ Dual-Form Decision Function
αᵢ — dual variable (Lagrange multiplier), with non-zero terms corresponding to support vectors
yᵢ — label of the training sample
K(xᵢ, x) — kernel function, computing the similarity between a training sample and a test sample
Interactive Simulation
Adjust the parameters on the left and click "Train SVM" to observe changes in the decision boundary, margin region, and support vectors.
Model Parameters
Training accuracy: --
Number of support vectors: --
Margin width: --
Final loss: --
Decision Boundary Visualization
About Support Vector Machine (SVM)
Standard Introduction
Support Vector Machine (SVM) is a supervised learning model based on statistical learning theory, formally proposed by Vapnik et al. in 1995. The core idea of SVM is to find an optimal hyperplane in feature space that separates samples of different classes, such that the distance from this hyperplane to the nearest training samples (i.e., support vectors)—called the margin—is maximized. For linearly inseparable data, SVM introduces kernel functions (such as the Gaussian RBF kernel and polynomial kernel) to implicitly map the data into a high-dimensional feature space, constructing a linear decision boundary in that high-dimensional space, thereby solving nonlinear problems. SVM has a solid mathematical foundation, based on VC dimension theory and the structural risk minimization principle, and possesses strong generalization ability, making it especially suitable for classification problems with small samples and high-dimensional data.
The SVM optimization problem can be stated as: under the constraints yᵢ(wᵀxᵢ + b) ≥ 1 - ξᵢ (ξᵢ ≥ 0), minimize the objective function (1/2)‖w‖² + C·Σξᵢ. Here C is the penalty coefficient, controlling the model's tolerance for misclassification. Through dual transformation, the problem becomes a quadratic programming problem that depends only on the inner products between samples (or kernel function values), and can be solved efficiently using algorithms such as SMO (Sequential Minimal Optimization). The final decision function is f(x) = ΣαᵢyᵢK(xᵢ, x) + b, in which only support vectors (samples with αᵢ > 0) participate in the decision, reflecting the sparsity advantage of SVM.
Plain-Language Introduction
Imagine you have two piles of balls of different colors (red and blue) on a plane, and you need to draw a line to separate them. What Support Vector Machine does is find the best line—not just any line that can separate them, but the one that makes the distance from the nearest balls to the line as large as possible. These balls closest to the line are the "support vectors," and like "support pillars," they determine the position of the line. If the balls are not linearly distributed (for example, red balls form a circle with blue balls inside), then you need to "project" the balls into a higher dimension (like bouncing points on a plane into the air), and then find a "hyperplane" in the high-dimensional space to separate them. The mathematical tool for this "projection" is the kernel function. The advantage of SVM is that it only cares about the points "on the boundary" (support vectors) rather than all data points, so it is computationally efficient and not prone to overfitting.
What Problem Does It Solve
- Binary classification problems: SVM was originally designed to solve binary classification problems by finding the optimal classification hyperplane to separate samples of different classes.
- Nonlinear classification: Through the kernel trick, SVM can handle linearly inseparable data and achieve nonlinear decision boundaries without explicitly computing high-dimensional mappings.
- Small-sample learning: Compared with traditional statistical methods (such as neural networks), SVM can still maintain good generalization performance in small-sample situations and is not prone to overfitting.
- High-dimensional data classification: SVM performs excellently when handling high-dimensional sparse data (such as word frequency vectors in text classification), because its complexity mainly depends on the number of support vectors rather than the feature dimension.
- Multi-class problems: Through strategies such as one-vs-one or one-vs-rest, SVM can be extended to multi-class tasks.
Why It Matters
- Solid theoretical foundation: SVM is based on statistical learning theory and the structural risk minimization principle, and its generalization error upper bound has rigorous mathematical guarantees, which gives SVM theoretical superiority.
- Global optimal solution: Unlike algorithms such as neural networks that easily fall into local optima, the SVM optimization problem is a convex quadratic programming problem, guaranteeing that a global optimal solution is found.
- Sparsity: The final model is determined only by a small number of support vectors, and the model complexity does not grow linearly with the number of training samples, resulting in high computational efficiency at prediction time.
- Generality of the kernel trick: The idea of kernel methods is not limited to SVM; it has also inspired many kernel methods such as kernel PCA and kernel regression, becoming an important branch of Machine Learning.
- Historical status: Before the rise of deep learning, SVM was one of the most mainstream classification algorithms in the Machine Learning field, and it remains competitive in many areas (such as bioinformatics and text classification) to this day.
Application Scenarios
- Text classification and sentiment analysis: SVM performs excellently in tasks such as spam filtering, news classification, and sentiment polarity judgment, and is especially suitable for high-dimensional sparse text features.
- Image recognition: For handwritten digit recognition (such as MNIST), face detection, and object classification, SVM is often used in combination with feature extraction methods such as HOG and SIFT.
- Bioinformatics: For gene expression data classification, protein structure prediction, and disease diagnosis, SVM can handle high-dimensional small-sample biological data.
- Financial risk control: For credit scoring, fraud detection, and stock trend prediction, the robustness of SVM has led to its wide application in the financial field.
- Medical diagnosis: For auxiliary disease diagnosis based on medical images or test data, such as cancer classification and pathological image analysis.
- Industrial inspection: For classification problems in industrial scenarios such as product quality inspection, fault diagnosis, and anomaly detection.
Limitations and Drawbacks
- Difficult training on large-scale data: The training time complexity of SVM ranges between O(n²) and O(n³) (where n is the number of samples). When the sample size exceeds hundreds of thousands, training efficiency drops significantly.
- Sensitive to parameter selection: The penalty coefficient C and kernel function parameters (such as γ of the RBF kernel) greatly affect model performance, and the optimal parameter combination usually needs to be determined through grid search or cross-validation.
- Kernel function selection relies on experience: Different problems may require different kernel functions, and there is a lack of unified theoretical guidance on how to choose a suitable kernel function, often requiring domain knowledge and experimental validation.
- Probability output is not direct: Standard SVM outputs the value of the decision function rather than a probability, and probability estimates require post-processing methods such as Platt scaling.
- Sensitive to class imbalance: When the ratio of positive to negative samples is severely imbalanced, the SVM decision boundary may be biased toward the minority class, requiring class weights or sampling strategies to mitigate.
- Relatively limited interpretability: Although support vectors provide an intuitive understanding of the decision boundary, when nonlinear kernel functions are used, the model's interpretability is less intuitive than methods such as decision trees.
- Not suitable for large-scale multi-class classification: Extending SVM to multi-class problems requires constructing multiple binary classifiers, which incurs high computational overhead in large-scale multi-class scenarios.