SimLabs Machine Learning Theory

Machine Learning Theory Visualization and Interactive Simulation

Model Evaluation and Selection
Empirical Error and Overfitting
Empirical error is the average loss of the model on the training set; overfitting means the model fits the training data too closely and weakens its generalization ability.
Evaluation Methods
Split the dataset through the hold-out method, cross-validation, or bootstrapping to objectively evaluate the model's generalization performance.
Performance Measures
Use metrics such as accuracy, precision, recall, and F1 score to quantify the model's predictive ability on the test set.
ROC Curve and AUC
The ROC curve depicts the trade-off between the true positive rate and the false positive rate, and the AUC value measures the overall performance of a classifier's ranking quality.
Cost-Sensitive Error Rate and Cost Curve
The cost-sensitive error rate assigns asymmetric costs to different types of misclassification, and the cost curve is used to visualize the expected cost under the optimal threshold.
Hypothesis Testing
Hypothesis testing uses statistical significance to determine whether a difference in model performance is caused by random fluctuation rather than a true performance difference.
t-Test
The t-test is used to compare whether there is a significant difference in the average performance of two models across multiple test sets.
McNemar's Test
McNemar's test analyzes the consistency difference between the predictions of two classifiers on the same test set based on a contingency table.
Friedman Test and Nemenyi Test
The Friedman test compares the average ranks of multiple models across multiple datasets, and the Nemenyi test performs post-hoc pairwise comparisons.
Bias and Variance
Bias measures the accuracy of the model's predictions, and variance measures the model's sensitivity to fluctuations in the training data; together they determine the generalization error.
Linear Models
Decision Trees
Neural Networks
Neuron Model
A basic computational unit that simulates biological neurons, receiving input signals and producing output through a weighted sum and nonlinear transformation.
M-P Neuron Model
The McCulloch-Pitts neuron, a binary threshold logic unit, is an early mathematical abstraction of artificial neural networks.
Activation Functions
Introduce nonlinear transformations (such as ReLU, Sigmoid, and Tanh) into neural networks, enabling the network to approximate arbitrarily complex functions.
Perceptron and Multilayer Networks
The perceptron is a single-layer linear classifier, and multilayer networks (MLPs) achieve nonlinear decision boundaries through hidden layers.
Backpropagation
Use the chain rule to compute the gradient of the loss function with respect to the weights; this is the core algorithm for training multilayer neural networks.
Global and Local Extrema
The global minimum and local minima in the loss function surface; optimization algorithms need to avoid getting trapped in local optima.
RBF Network
Radial basis function network, a single-hidden-layer feedforward neural network using Gaussian kernels as activation functions, skilled at local approximation.
ART Network
Adaptive Resonance Theory network, achieving incremental and stable pattern recognition through competitive learning and a vigilance mechanism.
SOM Network
Self-Organizing Map network, mapping high-dimensional input data onto a low-dimensional topological grid in an order-preserving way, achieving clustering and visualization.
Cascade-Correlation Network
The cascade-correlation network dynamically adds hidden units and maximizes their correlation with the residual, achieving structure-adaptive learning.
ELM Network
Extreme Learning Machine, which randomly initializes hidden layer weights and analytically computes the output weights, making training extremely fast.
Boltzmann Machine
A stochastic neural network based on an energy model that learns the data distribution through simulated annealing and is the foundation of Deep Belief Networks.
DBN
Deep Belief Network, composed of multiple stacked Restricted Boltzmann Machines, extracting features through layer-by-layer unsupervised pretraining.
CNN
Convolutional Neural Network, extracting local spatial features through convolution kernels, pooling layers, and shared weights, widely used in vision tasks.
Support Vector Machines
Bayesian Classifiers