Log loss

The loss function used in binary logistic regression.1

Log Loss=(x,y)Dylog(y)(1y)log(1y)\text{Log Loss} = \sum_{(x,y) \in D} -y \log(y') - (1-y) \log(1-y')

where

  • (x,y)D(x, y) \in D is the dataset containing many labeled examples, wich are (x,y)(x, y) pairs.
  • yy is the label in a labeled example. Since this is logistic regression, every value of yy must either be 0 or 1.
  • yy' is the predicted value (somewhere between 0 and 1, exclusive), given the set of features in xx.

Rationale

Squared loss works well for a linear regression where the rate of change of the output values is constant. However, the rate of change of a logistic regression model is not constant.

If you used squared loss to calculate errors for the sigmoid function, as the output got closer and closer to 0 and 1, you would need more memory to preserve the precision needed to track these values.

Instead, the loss function for logistic regression is log loss. The Log Loss equation returns the logarithm of the magnitude of the change, rather than just the distance from data to prediction.2

Footnotes

  1. developers.google.com/machine-learning/glossary#Log_Loss

  2. ML crash course - Logistic regression

2024 © ak