Skip to content
Search

Logistic regression

lastmod 2018-10-24created 2015-12-2811 edits

Logistic regression is a probabilistic model proposed by D.R.Cox in 1958, a statistical technique used to predict the likelihood of an event using a linear combination of independent variables.

As with regression analysis in general, the purpose of logistic regression is to express the relationship between the dependent and independent variables as a concrete function and use it in future prediction models. In the sense that it explains the dependent variable with a linear combination of independent variables, it is similar to linear regression analysis. However, unlike linear regression analysis, logistic regression targets categorical data as the dependent variable, and given input data, the result for that data is divided into a specific class, so it can also be seen as a kind of classification technique.

Logistic regression usually refers to problems where the dependent variable is binary (that is, the number of valid categories is two). Otherwise, when the problem has more than two categories, it is called multinomial logistic regression or polytomous logistic regression, and when there are multiple categories with an order, it is called ordinal logistic regression. Logistic regression analysis is widely used as a model for classification and prediction in various fields such as medicine, telecommunications, and data mining.

Pros and cons

  • Pros: Low computational cost, easy to implement, and easy knowledge representation for interpreting the results.
  • Cons: Tends toward underfitting, so accuracy may be low.
  • Applies to: numeric values, nominal values.

Pseudocode

The pseudocode for gradient ascent is as follows. 

Start with the weights all set to 1 // 모두 1로 설정된 가중치를 가지고 시작.
Repeat R number of times: // R회 반복.
    Calculate the gradient of the entire dataset // 입력 데이터 집합의 기울기를 계산.
    Update the weights vector by alpha * gradient // 알파 * 기울기로 가중치 벡터를 변경.
    Return the weights vector // 가중치 벡터 변환.

Example

If Regression deals with continuous values, Classification deals with discrete values. So examples of Classification are:

  • Whether an email is spam or not
  • Whether an online transaction is fraudulent or not (Online Transaction: Fraudulent)
  • Given the tumor size, whether the tumor is malignant or not

Tumor_size_-malignant-logistic_regression-_figure_2.png

In a case like the above, solving the problem with Regression may look right for now, but if a tumor appears at an odd position, the line changes significantly as shown below.

Tumor_size_-malignant-logistic_regression-_figure_1.png

So for discrete values like this, accuracy is higher if, rather than using Regression, you rely on a Threshold and predict y=1 when h(x) is above a certain value. Moreover, since regression is a straight line, for y where 0 <= y <= 1, it can produce y smaller than 0 or greater than 1.

For these reasons, Regression is rarely used for Classification problems. However, there is also Logistic Regression, where the range of y is 0 <= h(x) <= 1. It is sometimes used for Classification.

See also

Favorite site

author Zer0 sitemap