Overfitting

lastmod 2022-11-25created 2015-11-1216 edits

In statistics and machine learning, overfitting occurs when a statistical model describes random error or noise instead of the underlying relationship. Overfitting generally occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. A model that has been overfit will generally have poor predictive performance, as it can exaggerate minor fluctuations in the data.

About

• Data mining story 1: On overfitting (Overfitting)

One of the techniques mainly used in data mining is machine learning (Machine Learning). Machine learning refers to methods that go through a learning (Learning) process using past data to classify, estimate, predict, and so on for future cases. One of the problems that often occurs in machine learning is overfitting (Overfitting). In this post, I write a few words about overfitting.

In machine learning, the data set used for training is called the training data set (Training Data Set). To a machine learning algorithm, this training data set is the whole world. A machine learning algorithm tries its hardest to find the patterns or relationships contained in the training data set given to it. And so it finds the patterns contained in this data set. Then, will these patterns also fit well when applied to data sets that come later?

The answer is that they may or may not fit well. The reason is closely related to whether the training data set can really represent the entire population we are interested in. The training data set may contain the patterns that the entire population has. It may also miss some of them. But another problem is that the machine learning algorithm also learns characteristics that only the training data set has and the entire population does not. In other words, it studied too hard and ended up learning even unnecessary things. This phenomenon is called overfitting (Overfitting).

from the Gradient descent

To find a suitable Hypothesis, we can find the minimum of the Loss function with Gradient descent. Now it seems there is nothing more for us to worry about. No, actually that is not the case. According to our earlier assumption, we assumed that the dataset we have now represents the distribution of the entire dataset. But can we guarantee that this will necessarily be so? Of course not. The data we are looking at may be a tiny exception within a huge pattern, or because of noise, or too few samples, we may end up finding a pattern in the wrong direction. This case is called Overfitting. That is, the state of being so faithful to the existing data that the model is practically useless when new data comes in is called Overfitting. There are several ways to solve this, namely Model Selection and Regularization.

• Prev: Gradient descent

List of methods

• machine learning - Class 14 : Neural Network - Causes of slowdown in learning speed -

Solutions for preventing overfitting:

• Model Selection
• Regularization
• Early stopping
• Generalization
• Gradient descent
• Elastic distortion (by Microsoft)
• Affine transformation

See also

• Deep learning
• Backpropagation
• Occam's razor
• Learning rate

Favorite site

• Wikipedia (en) description of Overfitting
• [Recommended] Machine Learning Study (3) Overfitting
  • Requirement: Machine Learning Study (2) Probability Theory
• Machine Learning Chapter 9 - Learning rate, data preprocessing, overfitting -
• 10. Understanding and overcoming overfitting and underfitting (Regularization (lr) via bias, variance / Early Stop / Dropout)
• A chronic ailment of deep learning: tips for solving Overfitting!