Skip to content
Search

Regression analysis

lastmod 2026-07-12created 2017-02-2414 edits

In statistics, regression analysis (回歸分析) is an analysis method that, for observed continuous variables, derives a model between two variables and then measures its goodness of fit.

Regression analysis can be used for statistical prediction, such as data that changes over time, some effect, hypothetical experiments, and modeling causal relationships. However, in many cases it is used without properly establishing whether the assumptions hold, and the results are misused. In particular, advances in statistical software have made analysis easy and results easy to obtain, but it is up to the researcher to judge whether the analysis method chosen was appropriate and whether the information analysis is accurate.

Various types of regression analysis

Regression analysis not only tells you whether a significant relationship exists between two or more variables, but can also provide more specific information about that relationship. In particular, it can estimate the strength of the effect that several variables have on the dependent variable. If you change the value of one variable (say, price), regression analysis tells you how it will affect the dependent variable (sales).

Korean: 선형 회귀 (Linear regression)
Linear regression, arguably the most common type of regression analysis in machine learning, consists of a predictor variable and a dependent variable, which are related to each other in a linear way. As described above, linear regression uses the line that fits best, i.e. the line of best fit.
Use linear regression when the variables are linearly related to each other. An example would be predicting the effect of increased advertising spend on sales. However, linear regression is susceptible to outliers, so it should not be used to analyze big data sets.
Logistic regression (Logistic regression)
When the dependent variable has discrete values, in other words when it can take only one of two values such as 0 or 1, true or false, black or white, spam or not spam, you can analyze the data using logistic regression.
Logistic regression uses an S-shaped curve to show the relationship between the target variable and the independent variables. Note, however, that the logistic regression approach works best on large data sets where nearly equal values occur in the target variable. In this case, the data set should not include independent variables that are highly correlated with each other, because they can cause problems when ranking the variables. (This is the phenomenon known as multicollinearity, where some predictor variables of the model used in regression analysis are highly correlated with other predictor variables, which negatively affects data analysis.)
Ridge regression (Ridge regression)
However, if high correlation among the independent variables is unavoidable, ridge regression is a more suitable approach. Ridge regression, also called multiple regression, is known as a regularization technique and is used to reduce model complexity. It also uses a small amount of bias, known as the 'ridge regression penalty', to make the model less susceptible to overfitting.
Lasso regression (Lasso regression)
Lasso regression is another regularization technique that, like ridge regression, reduces model complexity. It reduces complexity by prohibiting the absolute size of the regression coefficients. Unlike ridge regression, it drives coefficient values all the way close to 0.
Its advantage is that it allows feature selection. You can build a model by selecting a feature set from the data set. Lasso regression can prevent overfitting by using only the necessary factors and setting the rest to 0.
Polynomial regression (Polynomial regression)
Polynomial regression models a nonlinear data set using a linear model. This is like fitting a square nail or peg into a round hole. Polynomial regression works in a way similar to multiple linear regression, meaning linear regression with several independent variables, but uses a nonlinear curve. In other words, it is used when the data points exist in a nonlinear way.
The model transforms these data points into polynomial features of a specified degree and models them using a linear model. It performs the best fit using a curved polynomial line rather than the straight line seen in linear regression. However, this model is prone to overfitting, so it is a good idea to analyze the curve at the ends to avoid strange results.

History

The original meaning of regression (English: regress) is to go back to a former state. The British geneticist Francis Galton, while studying the relationship between parents' heights and their children's heights, hypothesized that there is a linear relationship between the heights of parents and children and that, rather than getting taller or shorter, height tends to return to the overall average, and he called the method of analyzing this "regression analysis". Following this empirical research, Karl Pearson derived a functional relationship based on his survey of the heights of fathers and sons and established the theory of regression analysis mathematically.

What 'regresses' to where?

The dictionary definition of regression is 'to go around and return to the original place'. So to understand the meaning of 'regression' in a regression model, it is important to know what regresses to where. Many articles, including the one linked above, use Galton's 'regression to the mean' as an example and explain this regression as the data regressing to the trend line given by the regression model.

However, as the comments on the link above also point out, the explanation that the data regresses to the trend line feels somewhat awkward. If anything, the opposite, that the trend line follows the data, seems a more accurate explanation. That seems to be why some argue it should be called 'progression' rather than 'regression'. In my view, however, this explanation is not right.

As I understand it, 'regression' in a regression model means 'the residuals (residual: the difference between the observed value of the data and the predicted value of the model, i.e. the observed value of the error term in the regression equation) regress to the mean'. And in this context, a regression model can be defined as 'a model built so that the residuals regress to the mean'. Once you define it this way, you can see that the sentence below is inaccurate. 

회귀 모델링은 오차의 합이 최소가 되도록 만드는 작업이다.

If you describe building a regression model as finding 'the line with the smallest sum of errors against the data', that is, strictly speaking, only half right. When actually doing regression modeling, the following precondition must be met before finding the line that minimizes the residuals.

The residuals between the observed values of the data and the model's estimates must satisfy the i.i.d. (Independent and Identically Distributed random) property.

If the above precondition is not met, it is not a proper regression model even if it is the line that minimizes the error. To explain why, you first need to understand what modeling is.

See also

Favorite site

References

  1. Gimmesilver_-_What_is_regression.pdf ↩

author Zer0 sitemap