Skip to content
Search

Loss function

lastmod 2019-05-09created 2015-12-2129 edits

In statistics, decision theory, and economics, a loss function is a function that maps an event (technically, an element of a sample space) to a real number representing the economic loss associated with that event. It is also called a Cost function.

Cost Minimization

Expected loss

Since a loss function satisfies the definition of a random variable, its cumulative distribution function and expected value can be obtained.

The expected loss is

\[ \Lambda = \int_{-\infty}^\infty \!\!\lambda(x)\, f(x)\, \mathrm{d}x \]

where

  • λ(x) = loss function
  • x = continuous random variable
  • f(x)= probability density function.

In the Machine learning

How can we tell whether this Hypothesis is a good Hypothesis or not? The answer is actually simple: 'how well it predicts the results of new data'. But without new data, Scoring is impossible.1 So, assuming that the pattern found in the data we have now is the same as the overall pattern, it should be possible to Score by how well it Fits the current data. In other words, Scoring should be done with new data, but because we have to Score and use the Hypothesis before receiving new data, we assume the Dataset we have now is similar to the data that will come in later and do the Fitting with the values we have now.

This is where the Loss function comes in. It measures the Loss of the Hypothesis and minimizes that value. As the name Loss or Cost suggests, this value is computed from the difference between the value predicted (Predicted) by the function and the measured (Observed) value. There are various forms of Loss function; the simplest function to use is as follows.

\[ (predicted(x) - observed(x))^2 \]

There is also 0-1 Loss, a very Strict Loss function whose Loss is 0 if the values match exactly and 1 otherwise.

Comment

A function that computes the degree of difference between the output an artificial neural network predicts through computation and the output corresponding to the true answer. There are various loss functions; commonly used are RMSE (Root Mean Squared Error), which computes every error of each output neuron to express the degree of error, and the somewhat improved cross-entropy (Cross-Entorpy) error calculation.

Example

How do we pick the best one among the Hypothesis candidates? The concept introduced for this is the Cost function, or Loss function.

By defining a Cost function, we can infer how accurate the function is for the given data. For example, suppose our predicted function is y = x and (x, y) = (1, 2), (2, 3), (4, 5). Then when x is 1, 2, 3, the predicted value and the actual value will each differ by 1. The Cost function uses these differences to infer how close the function is to the original shape.

Cost function of linear regression

The Hypothesis of the basic cost function of Linear regression is as follows.

\[ H(x) = Wx + b \\ cost(W, b) = \frac{1}{m}\sum_{i=1}^{m}(H(x^{(i)}) - y^{(i)})^2 \] The reasons for the formula above are as follows.

  1. With plain subtraction, negative and positive values get mixed depending on the position of the line. The calculation becomes tedious.
  2. Taking the absolute value is easiest, but squaring is another option, since (negative x negative) and (positive x positive) always give a positive number
  3. When squared, nearby data gives small values and distant data gives large values, so a penalty can be imposed on distant data.

The goal is to find the W (slope) and b (intercept) that minimize the cost.

To simplify the hypothesis formula, b, which corresponds to the y-intercept, is removed.

Removing b to lighten the formula may seem odd, but once the number of W grows and things have to be handled with matrices, b is put into the matrix to keep the formula simple. It is a thoughtful touch that the value, which would disappear from the formula anyway, was removed in advance to help our understanding.

\[ H(x) = Wx \\ cost(W) = \frac{1}{m}\sum_{i=1}^{m}(Wx^{(i)} - y^{(i)})^2 \]

The conversion to the Formal definition is as follows.

\[ cost(W) = \frac{1}{2m}\sum_{i=1}^{m}(Wx^{(i)} - y^{(i)})^2 \] First, understand that m, used to take the mean of the squares, is applied after the calculation with W and b is done, so dividing by m or by 2m does not affect W and b.

The purpose of computing the cost is that we need the W and b at which the cost is minimal. What the minimum cost actually is does not matter at all. When there are several costs, find the smallest of them and use the W and b at that point. If all the costs are multiplied or divided by 2, the costs increase or decrease, but their relative sizes do not change.

Then why use 2m? Differentiating a square brings a 2 to the front, and multiplying the 2 in the numerator by the 2 in the denominator to make 1 simplifies the formula, so it is put in deliberately.

The formula for determining the next position of the slope (W) is as follows.

W := W - change in W

To find the amount of change, take the derivative (Deravative).

\[ W := W - \alpha\frac{\partial}{\partial W}cost(W) \] α (alpha) is a constant we add arbitrarily and does not take part in the differentiation. If α (alpha) is too large, overshooting can occur, and if it is too small, it takes too long to converge to the minimum cost. This value is an important one that must be determined through repeated testing, but again it does not take part in the differentiation.

The result of the differentiation is as follows.

\[ \begin{align} W &:= W - \alpha\frac{\partial}{\partial W}cost(W) \\ &:= W - \alpha\frac{\partial}{\partial W} \frac{1}{2m}\sum_{i=1}^{m}(Wx^{(i)} - y^{(i)})^2 \\ &:= W - \alpha\frac{\partial}{\partial W} \frac{1}{2m}\sum_{i=1}^{m}(Wx^{(i)} - y^{(i)})(Wx^{(i)} - y^{(i)}) \\ &:= W - \alpha\frac{\partial}{\partial W} \frac{1}{2m}\sum_{i=1}^{m}(Wx^{(i)})^2 - 2Wx^{(i)}y^{(i)} + (y^{(i)})^2 \\ &:= W - \alpha\frac{1}{2m}\sum_{i=1}^{m} \frac{\partial}{\partial W}( W^{2}(x^{(i)})^2 - 2Wx^{(i)}y^{(i)} + (y^{(i)})^2 ) \\ &:= W - \alpha\frac{1}{2m}\sum_{i=1}^{m} 2W(x^{(i)})^2 - 2x^{(i)}y^{(i)} \\ &:= W - \alpha\frac{1}{2m}\sum_{i=1}^{m} 2(Wx^{(i)} - y^{(i)})x^{(i)} \\ &:= W - \alpha\frac{1}{m}\sum_{i=1}^{m} (Wx^{(i)} - y^{(i)})x^{(i)} \\ \end{align} \]

Effective Loss function

See also

Favorite site

References

  1. It is not entirely impossible; there is a Skill for doing Validation through Model selection. ↩

author Zer0 sitemap