Skip to content
Search

Precision and recall

lastmod 2026-10-15created 2018-12-0518 edits

In pattern recognition and information retrieval using binary classification, precision is the fraction of retrieved results that are classified as relevant, and recall is the fraction of items classified as relevant that are actually retrieved. Both precision and recall are therefore based on a measure and understanding of relevance.

mAP (Mean Average Precision)

mAP is a metric built on precision and recall, the performance metrics of binary classifiers, and is widely used to evaluate the performance of object detection algorithms.

  • Precision is the fraction of positive predictions that are actually true. It is defined as ‘true positives / (true positives + false positives)’.
  • Recall is the fraction of actually true cases that are predicted positive. It is defined as ‘true positives / (true positives + false negatives)’.

\[ \text{Precision} = \frac{TP}{TP + FP}, \qquad \text{Recall} = \frac{TP}{TP + FN} \]

Recall is also called sensitivity.

Judging detections as true or false

A bounding box output by object detection carries only a location and a confidence score; it does not say whether the box is true or false. Whether a bounding box is counted as true or false is decided by checking whether the IoU between the ground truth and the detected box is above a certain value.

\[ \text{IoU}(B_p, B_{gt}) = \frac{|B_p \cap B_{gt}|}{|B_p \cup B_{gt}|} \ge \tau \]

  • \(B_p\): detected bounding box, \(B_{gt}\): ground truth box
  • \(\tau\): IoU threshold. PASCAL VOC uses 0.5.
  • Detections are matched in order of decreasing confidence, and only one detection per ground-truth object counts as a TP. A box that detects an already matched object again is counted as an FP.

AP (Average Precision)

Deep learning object detection algorithms also output a confidence score indicating how likely a detected box is to be true. The boxes judged positive change depending on the ‘threshold’, the cutoff confidence above which a detection is judged positive, so precision and recall both change as well. If you lower the threshold from high to low and measure precision and recall each time a truly true object is newly detected (TP), you get as many precision/recall pairs as there are detected true objects. Taking only the precision values and averaging them gives the average precision (AP). Ground-truth objects that are never detected are counted as precision 0 and included in the average.

Sort the \(n\) detections by decreasing confidence, and set \(\text{rel}(k) = 1\) if the \(k\)-th detection is a TP and \(\text{rel}(k) = 0\) if it is an FP. With the threshold set to the confidence of the \(k\)-th detection, precision and recall are:

\[ P(k) = \frac{1}{k} \sum_{i=1}^{k} \text{rel}(i), \qquad R(k) = \frac{1}{N_{gt}} \sum_{i=1}^{k} \text{rel}(i) \]

AP is the average of the precision measured at each TP (\(\text{rel}(k) = 1\)).

\[ \text{AP} = \frac{1}{N_{gt}} \sum_{k=1}^{n} P(k)\, \text{rel}(k) \]

  • \(N_{gt}\): number of ground-truth objects
  • Because the denominator is \(N_{gt}\), not the number of detected TPs, objects that are never detected enter the average with precision 0.

Since \(\Delta R(k) = R(k) - R(k-1) = \text{rel}(k) / N_{gt}\), AP equals the area under the precision-recall curve (PR curve).

\[ \text{AP} = \sum_{k=1}^{n} P(k)\, \Delta R(k) \approx \int_{0}^{1} p(r)\, dr \]

Interpolated AP

Because the PR curve zigzags, benchmarks usually make the curve monotonically decreasing by taking the maximum precision at each recall level or higher, and then compute AP.

\[ p_{\text{interp}}(r) = \max_{\tilde{r} \ge r} p(\tilde{r}) \]

11-point interpolation (PASCAL VOC 2007): averages the interpolated precision at 11 recall points 0, 0.1, …, 1.0.

\[ \text{AP}_{11} = \frac{1}{11} \sum_{r \in \{0,\, 0.1,\, \dots,\, 1.0\}} p_{\text{interp}}(r) \]

All-point interpolation (PASCAL VOC 2010 and later): computes the area using the interpolated precision at every point \(r_k\) (\(r_0 = 0\)) where recall changes.

\[ \text{AP} = \sum_{k} (r_{k+1} - r_k)\, p_{\text{interp}}(r_{k+1}) \]

COCO uses 101 recall points 0, 0.01, …, 1.0.

mAP

mAP is the average precision averaged once more. Compute the average precision on a fixed test set for each class the classifier can detect, then average those average precisions again to get the mAP.

\[ \text{mAP} = \frac{1}{|C|} \sum_{c \in C} \text{AP}_c \]

  • \(C\): set of classes
  • \(\text{AP}_c\): AP of class \(c\)

Substituting the AP formula gives it in one line:

\[ \text{mAP} = \frac{1}{|C|} \sum_{c \in C} \frac{1}{N_{gt}^{(c)}} \sum_{k=1}^{n_c} P_c(k)\, \text{rel}_c(k) \]

The result depends on the IoU threshold \(\tau\), so it is usually written alongside. For example, [email protected] is the mAP at \(\tau = 0.5\). COCO averages the 10 mAPs obtained by varying \(\tau\) from 0.50 to 0.95 in steps of 0.05, written as mAP@[.5:.95]. (The COCO documentation simply calls this value AP.)

\[ \text{mAP}_{0.5:0.95} = \frac{1}{10} \sum_{\tau \in \{0.50,\, 0.55,\, \dots,\, 0.95\}} \text{mAP}_{\tau} \]

Information retrieval

In information retrieval, precision and recall are defined in terms of retrieved documents (documents produced by a web search engine) and relevant documents (web documents actually relevant to the topic being searched for).

Precision

In information retrieval, precision is the fraction of retrieved documents that are relevant.

\[ \text{precision}=\frac{|\{\text{relevant documents}\}\cap\{\text{retrieved documents}\}|}{|\{\text{retrieved documents}\}|} \]

Recall

In information retrieval, recall is the fraction of relevant documents that are actually retrieved.

\[ \text{recall}=\frac{|\{\text{relevant documents}\}\cap\{\text{retrieved documents}\}|}{|\{\text{relevant documents}\}|} \]

Confusion matrix representation

Actual answer

Positive

Negative

Test result

Positive

True Positive

False Positive</br>(Type I error)

Negative

False Negative</br>(Type II error)

True Negative

In statistical classification, precision and recall are defined as follows:

\(\text{Precision}=\frac{tp}{tp+fp} \,\)
\(\text{Recall}=\frac{tp}{tp+fn} \,\)

Here recall is also called Sensitivity, and precision is also called Positive Predictive Value (PPV); other measures used in statistical classification include True Negative Rate (Specificity) and Accuracy.

\(\text{True negative rate}=\frac{tn}{tn+fp} \,\)
\(\text{Accuracy}=\frac{tp+tn}{tp+tn+fp+fn} \,\)

APH (Average Precision weighted by Heading)

APH is a 3D Object Detection evaluation metric introduced in the Waymo Open Dataset benchmark. It weights the standard AP by the accuracy of each object's heading (heading/yaw). It was introduced because, in autonomous driving, which direction a vehicle or pedestrian is facing is critical for trajectory prediction.

Key idea

Standard AP evaluates only the box's position and size, but in autonomous driving, even at the right position, a wrong heading estimate throws trajectory prediction completely off. To cover this limitation, APH multiplies each True Positive by a weight based on its heading error.

Calculation

Weight for each True Positive based on the heading error \(\Delta\theta\):

\(w = 1 - \frac{\min(|\Delta\theta|,\ 2\pi - |\Delta\theta|)}{\pi}\)

  • Heading matches exactly (\(\Delta\theta = 0\)) → weight 1 (same as AP)
  • Heading off by 90° → weight 0.5
  • Heading opposite by 180° → weight 0 (not counted as a TP)

Therefore the following always holds:

\(\text{APH} \leq \text{AP}\)

The larger the difference between the two (AP − APH gap), the more inaccurate the model's heading estimation.

Waymo evaluation difficulty

The Waymo Open Dataset evaluates at two LEVELs:

Difficulty

Condition

Description

LEVEL_1

≥ 5 LiDAR points in the box

Easy (nearby, clearly visible objects)

LEVEL_2

≥ 1 LiDAR point in the box

Hard (includes distant and occluded objects)

Example evaluation result 

# Waymo evaluation result example / Waymo 평가 결과 예시
# LEVEL_2 vehicle detection
{
    "AP":  0.752,   # Position only / 위치 정확도만
    "APH": 0.741,   # With heading / 방향까지 고려
    # The smaller the AP-APH gap, the more accurate heading estimation
    # AP-APH 차이가 작을수록 heading 추정이 정확함
}

See also

Favorite site

References

  1. Darkpgmr.tistory.com_-_Understanding_precision_and_recall.pdf ↩

  2. Tykimos.github.io_-_Evaluation_Talk.pdf ↩

author Zer0 sitemap