Cross validation

lastmod 2019-02-12created 2015-12-2910 edits

Cross Validation is a method that uses part of the database as the Training set, but performs training and test while changing which part is used.

• An algorithm built using only the Training set works well on the Training set, but it is hard to judge it as a good Hypothesis Function
• It may show a low error rate on the Training set but a high error rate on other data
• To solve the problem above, part of the full training data is used as a Cross Validation Set
  • Training set (60%)
  • Cross Validation Set (20%)
  • Test set (20%)

K-fold cross-validation

One iteration of the K-fold cross-validation is performed in the following way: First, a random permutation of the sample set is generated and partitioned into K subsets ("folds") of about equal size. Of the K subsets, a single subset is retained as the validation data for testing the model (this subset is called the "testset"), and the remaining K - 1 subsets together are used as training data ("trainset"). Then a model is trained on the trainset and its accuracy is evaluated on the testset. Model training and evaluation is repeated K times, with each of the K subsets used exactly once as the testset.

The case of a 5-fold cross-validation with 30 samples is illustrated in the picture below:

Leave-one-out cross-validation

As the name suggests, leave-one-out cross-validation involves using a single sample from the original sample set as the validation data, and the remaining samples as the training data. This is repeated such that each sample in the sample set is used exactly once as the validation data. This is the same as K-fold cross-validation where K is equal to the number of samples in the sample set.

There is no need in generating random permutations for leave-one-out cross-validation and in repeating it, because the training and validation datasets for each of the folds are always the same, and therefore the result of the accuracy estimation is determined.

See also

• Machine learning
• Dataset

Favorite site

• Wikipedia (en) Description of Cross-validation
• [Recommended] Cross-Validation Explained
• [Recommended] Model evaluation, model selection, and algorithm selection in machine learning – Chapter 3. Cross-validation and hyperparameter tuning
• Machine Learning : How to select a model. 1. Cross validation
• Cross Validation
• Machine learning basics ( Machine Learning Basics ) :: Modulabs tech blog
• Evaluating a Hypothesis Function
• 5.1 Cross-Validation
• Advice for applying machine learning – Model selection and training/validation/test sets
• Data Science School - Cross-validation