Model selection
lastmod 2018-08-17created 2015-12-245 edits
Model selection is the task of selecting a statistical model from a set of candidate models, given data. In the simplest cases, a pre-existing set of data is considered. However, the task can also involve the design of experiments such that the data collected is well-suited to the problem of model selection. Given candidate models of similar predictive or explanatory power, the simplest model is most likely to be the best choice.
Overview
However, no matter how good a prior we put in or how good a penalty term we design, problems can arise if we do learning with flawed data. Like the blind men touching the elephant, what if the full data is distributed very widely but the data we have covers only a tiny part of it, or noise suddenly spikes in that situation and the data comes in entirely wrong? Learning with such data would likely cause sample bias and greatly degrade performance. In fact, the word complexity above is more precisely 'high complexity relative to the number of data points'. To avoid this as much as possible, we do something called generalization; in other words, we try to make our solution give general results rather than only specific ones. There can be many ways to do this, but the most commonly used method is to use something called a Validation set. That is, rather than using all the data for training, only part of it is used for training and the rest is used as a kind of validation check. If our model is a fairly good model and the validation set is well chosen, the results on the validation set should be as good as on the training set.
Usually the ratio of training to validation in the whole data is 8:2. But if the Validation set is too small, even this may not be enough to give good results. One concept that can solve this is Cross validation, which, rather than having just one validation set, defines several validation sets and does learning for each set. For example, given data X={1,2,3,4,5,6,7,8,9,10}, the first learning could use {1,..,8}, the next {2,..,9}, and so on, doing learning for every permutation. Naturally, learning several times this way gives a different model parameter each time, and cross-validation uses those values appropriately to obtain the most suitable parameter. You can use the average, the median, or various other methods. The disadvantage of cross-validation is that the algorithm's running time depends not only on the size of the data but also on the many sets used for validation. And as the data grows, the number of such validation sets becomes enormous. Precisely, it grows exponentially, so it is impossible to simply do cross-validation over all the data.
For this reason, when actually doing Cross validation, not all the data is used; instead, a few sets are picked and the model parameter is obtained 'roughly' several times. Of course, theoretically there are also concepts such as AIC and BIC, and methods of choosing a model according to them (AIC is the Akaike Information Criterion and BIC is the Bayesian Information Criterion; both are about which 'information criteria' to use.)