Multinomial logistic regression

lastmod 2018-08-17created 2015-12-2114 edits

In statistics, multinomial logistic regression is a classification method that generalizes logistic regression to multiclass problems, i.e. with more than two possible discrete outcomes. That is, it is a model that is used to predict the probabilities of the different possible outcomes of a categorically distributed dependent variable, given a set of independent variables (which may be real-valued, binary-valued, categorical-valued, etc.).

Multinomial logistic regression is known by a variety of other names, including polytomous LR, multiclass LR, softmax regression, multinomial logit, maximum entropy (MaxEnt) classifier, conditional maximum entropy model.

Softmax function

• Wikipedia (en) description of the Softmax function
• TensorFlow - (4) MNIST - Softmax Regression (Korean)
  • CHAPTER 3: Improving the way neural networks learn
• UFLDL Tutorial: Softmax Regression
• UFLDL Resources: Softmax Regression
• DEEP LEARNING TUTORIAL – SOFTMAX REGRESSION
• Softmax vs. Softmax-Loss: Numerical Stability (Chinese)
• Softmax layer in a neural network

In mathematics, in particular probability theory and related fields, the softmax function, or normalized exponential, is a generalization of the logistic function that "squashes" a K-dimensional vector \(\mathbf{z}\) of arbitrary real values to a K-dimensional vector \(\sigma(\mathbf{z})\) of real values in the range (0, 1) that add up to 1. The function is given by

\[ \sigma(\mathbf{z})_j = \frac{e^{z_j}}{\sum_{k=1}^K e^{z_k}} \] for j = 1, ..., K.

Artificial neural networks

In neural network simulations, the softmax function is often implemented at the final layer of a network used for classification. Such networks are then trained under a log loss (or cross-entropy) regime, giving a non-linear variant of multinomial logistic regression.

Since the function maps a vector and a specific index i to a real value, the derivative needs to take the index into account:

\[ \frac{\partial}{\partial q_k}\sigma(\textbf{q}, i) = \dots = \sigma(\textbf{q}, i)(\delta_{ik} - \sigma(\textbf{q}, k)) \]

Here, the Kronecker delta is used for simplicity (cf. the derivative of a sigmoid function, being expressed via the function itself).

See Multinomial logit for a probability model which uses the softmax activation function.

See also

• Logistic regression
• Machine learning
• Activation function
• Logistic function
• Logit

Favorite site

• Wikipedia (en) description of Multinomial logistic regression
• Deep learning tutorial 1. Logistic regression
• Softmax Regression