Evaluation Methods · Model Selection and Error Estimation
Hold-Out Method · Cross-Validation · Bootstrapping — Drag the sliders to adjust parameters and observe changes in dataset splitting and error
📂 Hold-Out Method
Standard Introduction: The hold-out method directly divides the dataset D into two mutually exclusive sets: the training set S and the test set T, that is, D = S ∪ T, S ∩ T = ∅. After training the model on S, T is used to evaluate the test error as an estimate of the generalization error. Stratified sampling is usually used to preserve class proportions. The estimate from a single split is not stable enough, so multiple random splits are generally averaged.
Plain-Language Introduction: It is like a teacher creating exam questions by splitting the question bank into "regular homework problems" and "final exam problems." Getting all the homework right does not mean you will do well on the final exam, so unseen final exam questions are used to test your true level. The hold-out method simply reserves a portion of the data as the "final exam" and uses the rest for practice.
Cross-Validation (k-fold Cross Validation)
Standard Introduction: The dataset D is divided into k mutually exclusive subsets of similar size. Each time, the union of k‑1 subsets is used as the training set and the remaining subset is used as the test set; a total of k training/testing rounds are performed, and finally the mean of these k test errors is returned. The commonly used version is 10-fold cross-validation. The special case k = |D| is called leave-one-out.
Plain-Language Introduction: Divide the question bank into k equal parts, and take turns using one part as a "quiz" while the remaining k‑1 parts are used for review. In this way, every question is tested once, and the final score is averaged, which is fairer and more stable than a single exam.
Bootstrapping
Standard Introduction: Based on the bootstrap sampling method, from a dataset D containing m samples, m samples are randomly drawn with replacement to obtain the training set D'. Some samples will appear repeatedly in D', while about 36.8% of the samples are never sampled and can be used as the test set (Out-of-Bag, OOB). Bootstrapping is useful when the dataset is small and it is difficult to effectively split it into training/test sets, but it changes the data distribution. A commonly used .632 estimate is:
Plain-Language Introduction: Draw one card at a time from a stack of cards, note the number, and then put it back, repeating this m times. Some cards will be drawn multiple times (the training set), while about 36.8% of the cards are never drawn at all (the test set). This method can "conjure up" a test set when there is very little data, but because of duplication, the model may be overoptimistic, so the .632 formula is used to correct the error.