SimLabs Machine Learning Theory
Machine Learning Theory Visualization and Interactive Simulation
Model Evaluation and Selection
Empirical Error and Overfitting
Empirical error is the average loss of the model on the training set; overfitting means the model fits the training data too closely and weakens its generalization ability.
Evaluation Methods
Split the dataset through the hold-out method, cross-validation, or bootstrapping to objectively evaluate the model's generalization performance.
Performance Measures
Use metrics such as accuracy, precision, recall, and F1 score to quantify the model's predictive ability on the test set.
ROC Curve and AUC
The ROC curve depicts the trade-off between the true positive rate and the false positive rate, and the AUC value measures the overall performance of a classifier's ranking quality.
Cost-Sensitive Error Rate and Cost Curve
The cost-sensitive error rate assigns asymmetric costs to different types of misclassification, and the cost curve is used to visualize the expected cost under the optimal threshold.
Hypothesis Testing
Hypothesis testing uses statistical significance to determine whether a difference in model performance is caused by random fluctuation rather than a true performance difference.
t-Test
The t-test is used to compare whether there is a significant difference in the average performance of two models across multiple test sets.
McNemar's Test
McNemar's test analyzes the consistency difference between the predictions of two classifiers on the same test set based on a contingency table.
Friedman Test and Nemenyi Test
The Friedman test compares the average ranks of multiple models across multiple datasets, and the Nemenyi test performs post-hoc pairwise comparisons.
Bias and Variance
Bias measures the accuracy of the model's predictions, and variance measures the model's sensitivity to fluctuations in the training data; together they determine the generalization error.
Linear Models
Simple Linear Regression Model
The most basic regression model for continuous value prediction through the linear relationship between a single independent variable and a dependent variable.
Multiple Linear Regression Model
Establish a regression equation using the linear relationship between multiple feature variables and a target variable for multidimensional prediction.
Logistic Regression Model
Map the output of linear regression into probability space through the Sigmoid function, used for binary classification tasks.
Linear Discriminant Analysis (LDA)
Find the optimal projection direction by maximizing the ratio of between-class scatter to within-class scatter, achieving classification and dimensionality reduction.
Multi-Class Models
Extend binary classifiers to handle multi-class classification problems through OvO, OvR, or softmax approaches.
Class Imbalance Handling (Rescaling)
Mitigate the problem of models favoring the majority class due to uneven class distribution through resampling or cost adjustment.
Decision Trees
Decision Tree Model
Perform feature partitioning based on a tree structure, building interpretable classification or regression models through recursive splitting.
Decision Tree Pruning
Reduce the risk of overfitting and improve the generalization ability of decision trees by removing redundant branches or subtrees.
Multivariate Decision Tree
Use a linear combination of multiple features for splitting at each node, enhancing the expressive power of the decision tree.
Neural Networks
Neuron Model
A basic computational unit that simulates biological neurons, receiving input signals and producing output through a weighted sum and nonlinear transformation.
M-P Neuron Model
The McCulloch-Pitts neuron, a binary threshold logic unit, is an early mathematical abstraction of artificial neural networks.
Activation Functions
Introduce nonlinear transformations (such as ReLU, Sigmoid, and Tanh) into neural networks, enabling the network to approximate arbitrarily complex functions.
Perceptron and Multilayer Networks
The perceptron is a single-layer linear classifier, and multilayer networks (MLPs) achieve nonlinear decision boundaries through hidden layers.
Backpropagation
Use the chain rule to compute the gradient of the loss function with respect to the weights; this is the core algorithm for training multilayer neural networks.
Global and Local Extrema
The global minimum and local minima in the loss function surface; optimization algorithms need to avoid getting trapped in local optima.
RBF Network
Radial basis function network, a single-hidden-layer feedforward neural network using Gaussian kernels as activation functions, skilled at local approximation.
ART Network
Adaptive Resonance Theory network, achieving incremental and stable pattern recognition through competitive learning and a vigilance mechanism.
SOM Network
Self-Organizing Map network, mapping high-dimensional input data onto a low-dimensional topological grid in an order-preserving way, achieving clustering and visualization.
Cascade-Correlation Network
The cascade-correlation network dynamically adds hidden units and maximizes their correlation with the residual, achieving structure-adaptive learning.
ELM Network
Extreme Learning Machine, which randomly initializes hidden layer weights and analytically computes the output weights, making training extremely fast.
Boltzmann Machine
A stochastic neural network based on an energy model that learns the data distribution through simulated annealing and is the foundation of Deep Belief Networks.
DBN
Deep Belief Network, composed of multiple stacked Restricted Boltzmann Machines, extracting features through layer-by-layer unsupervised pretraining.
CNN
Convolutional Neural Network, extracting local spatial features through convolution kernels, pooling layers, and shared weights, widely used in vision tasks.
Support Vector Machines
Bayesian Classifiers
Bayesian Decision Theory
Minimize the expected risk based on posterior probabilities and a loss function; this is the core framework of statistical decision theory.
Maximum Likelihood Estimation
Estimate unknown parameters by maximizing the likelihood function of the observed data under given parameters.
Naive Bayes Classifier
Based on Bayes' theorem and the conditional independence assumption of features, compute posterior probabilities and make classification decisions.