Bias–variance tradeoff
lastmod 2018-08-17created 2015-12-294 edits
In statistics and machine learning, the bias-variance tradeoff (or dilemma) is the problem faced when minimizing two kinds of error (bias and variance) in order to prevent a supervised learning algorithm from overgeneralizing beyond the scope of the training set.
- Bias is the error that occurs when a learning algorithm makes wrong assumptions. High bias causes the underfitting problem, in which the algorithm misses the relevant relations between the features of the data and the outputs.
- Variance is the error that occurs because of small fluctuations inherent in the training set. High variance causes the overfitting problem, in which even large noise gets included in the model.
The bias-variance decomposition is one way of analyzing the expected error of a learning algorithm; it views the error as the sum of bias, variance, and an error that is inherent in the data itself and cannot be reduced by any modeling. The bias-variance tradeoff applies to all forms of supervised learning, including classification, regression analysis, and structured output learning (structed output learning). It is also mentioned to explain the effectiveness of heuristics (intuitive judgment errors) in human learning.
About
The bias-variance trade-off is actually a very well-known concept. Simply put, it means that bias and variance have an unavoidable trade-off relationship. For the detailed proof, refer to the linked Wikipedia page. To mention only the concept briefly, bias is about how far, on average, our hypothesis h(x) is from the actual phenomenon y(x). That is, the lower the bias, the closer it is to the actual phenomenon, and the higher the bias, the farther it is from the actual phenomenon. This is very close to what the word bias actually means. Variance means how far things are from the mean: the higher the variance, the farther the data are from the mean, and the lower the variance, the closer the data are distributed around the mean. The interesting point is that the two have a complete trade-off relationship, and the idea of the proof is that when you compute the expected squared error of h(x) and y(x), the value decomposes exactly into expressions of bias and variance. For given data, if bias is low, variance is high, and if bias is high, variance is low. Here, bias becomes a term about overfitting or generalization, because low bias on the currently observed data means the model is too focused on just the current data. In other words, too low a bias can be said to mean almost the same as overfitting. Not only that, high variance can also be taken to mean high complexity, so too low a bias or a high variance should be avoided. But high bias is not always the answer either, because if you are not careful you may end up with an underfitted model whose bias is too high. In other words, bias can be said to be a measure of how much I trust the data I am looking at now, and if I do not trust the data I am looking at at all, it will be hard to expect a model obtained through machine learning to work well.