SSD:Paper

We present a method for detecting objects in images using a single deep neural network. Our approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes over different aspect ratios and scales per feature map location. At prediction time, the network generates scores for the presence of each object category in each default box and produces adjustments to the box to better match the object shape. Additionally, the network combines predictions from multiple feature maps with different resolutions to naturally handle objects of various sizes. SSD is simple relative to methods that require object proposals because it completely eliminates proposal generation and subsequent pixel or feature resampling stages and encapsulates all computation in a single network. This makes SSD easy to train and straightforward to integrate into systems that require a detection component. Experimental results on the PASCAL VOC, COCO, and ILSVRC datasets confirm that SSD has competitive accuracy to methods that utilize an additional object proposal step and is much faster, while providing a unified framework for both training and inference. For 300 × 300 input, SSD achieves 74.3% mAP1 on VOC2007 test at 59 FPS on a Nvidia Titan X and for 512 × 512 input, SSD achieves 76.9% mAP, outperforming a comparable state-of-the-art Faster R-CNN model. Compared to other single stage methods, SSD has much better accuracy even with a smaller input image size. Code is available at: https://github.com/weiliu89/caffe/tree/ssd .

We present a method for detecting objects in images using a single deep neural network. The approach, named SSD, discretizes the output space of bounding boxes into a set of default boxes with different aspect ratios and scales per feature map location. At prediction time, the network generates a score for the presence of each object category in each default box and adjusts the box to better match the object shape. In addition, the network combines predictions from multiple feature maps of different resolutions to naturally handle objects of various sizes. SSD is simple compared to methods that require object proposals, because it completely eliminates proposal generation and the subsequent pixel or feature resampling stage and encapsulates all computation in a single network. This makes SSD easy and straightforward to integrate into systems that require a detection component. Experimental results on the PASCAL VOC, COCO, and ILSVRC datasets show that SSD has superior accuracy compared to methods that use an additional object proposal step and is much faster, while providing a unified framework for both training and inference. For 300 × 300 input, SSD achieves 74.3 % mAP1 on the VOC2007 test at 59 FPS on an NVIDIA Titan X, and for 512x512 input, SSD achieves 76.9 % mAP, outperforming the latest FAST R-CNN model . Compared to other single-stage methods, SSD is much more accurate than when the input image size is small. The code is available at https://github.com/weiliu89/caffe/tree/ssd.

Current state-of-the-art object detection systems are variants of the following approach: hypothesize bounding boxes, resample pixels or features for each box, and apply a highquality classifier. This pipeline has prevailed on detection benchmarks since the Selective Search work [1] through the current leading results on PASCAL VOC, COCO, and ILSVRC detection all based on Faster R-CNN[2] albeit with deeper features such as [3]. While accurate, these approaches have been too computationally intensive for embedded systems and, even with high-end hardware, too slow for real-time applications. Often detection speed for these approaches is measured in seconds per frame (SPF), and even the fastest high-accuracy detector, Faster R-CNN, operates at only 7 frames per second (FPS). There have been many attempts to build faster detectors by attacking each stage of the detection pipeline (see related work in Sec. 4), but so far, significantly increased speed comes only at the cost of significantly decreased detection accuracy.

Current state-of-the-art object detection systems are variants of the following approach: hypothesize bounding boxes, resample pixels or features for each box, and apply a high-quality classifier. This pipeline has prevailed on detection benchmarks since the Selective Search work [1], through the latest results on PASCAL VOC, COCO, and ILSVRC detection, which are based on Fast R-CNN [2] albeit with deeper features such as [3]. While accurate, these approaches have been too computationally intensive for embedded systems and too slow for real-time applications even on high-end hardware. Detection speed for these approaches is often measured in seconds per frame (SPF), and even the fastest high-accuracy detector, Faster R-CNN, runs at 7 frames per second (FPS). There have been many attempts to build faster detectors by attacking each stage of the detection pipeline (see related work in Sec. 4), but so far speed has increased significantly only when detection accuracy dropped markedly.

This paper presents the first deep network based object detector that does not resample pixels or features for bounding box hypotheses and and is as accurate as approaches that do. This results in a significant improvement in speed for high-accuracy detection (59 FPS with mAP 74.3% on VOC2007 test, vs. Faster R-CNN 7 FPS with mAP 73.2% or YOLO 45 FPS with mAP 63.4%). The fundamental improvement in speed comes from eliminating bounding box proposals and the subsequent pixel or feature resampling stage. We are not the first to do this (cf [4,5]), but by adding a series of improvements, we manage to increase the accuracy significantly over previous attempts. Our improvements include using a small convolutional filter to predict object categories and offsets in bounding box locations, using separate predictors (filters) for different aspect ratio detections, and applying these filters to multiple feature maps from the later stages of a network in order to perform detection at multiple scales. With these modifications—especially using multiple layers for prediction at different scales—we can achieve high-accuracy using relatively low resolution input, further increasing detection speed. While these contributions may seem small independently, we note that the resulting system improves accuracy on real-time detection for PASCAL VOC from 63.4% mAP for YOLO to 74.3% mAP for our SSD. This is a larger relative improvement in detection accuracy than that from the recent, very high-profile work on residual networks [3]. Furthermore, significantly improving the speed of high-quality detection can broaden the range of settings where computer vision is useful.

This paper presents the first deep network based object detector that does not resample pixels or features for bounding box hypotheses, and it is as accurate as approaches that do. The result is a significant improvement in speed for high-accuracy detection (mAP 74.3 % on the VOP2007 test, versus Faster R-CNN at 7 FPS with mAP 73.2 % or YOLO at 45 FPS with mAP 63.4 %). The fundamental improvement in speed comes from eliminating bounding box proposals and the subsequent pixel or feature resampling stage. We are not the first to do this (cf [4,5]), but by adding a series of improvements we can significantly increase accuracy over previous attempts. Our improvements include using small convolutional filters to predict object categories and offsets in bounding box locations, using separate predictors (filters) for different aspect ratio detections, and applying these filters in the later stages of the network to perform detection at multiple scales. With these modifications (especially using multiple layers for prediction at multiple scales), we can increase detection speed with high accuracy using relatively low-resolution input. While these contributions may seem independent, the resulting system improves real-time detection accuracy on PASCAL VOC from 63.4 % for YOLO to 74.3 % for SSD. This is a larger relative improvement in detection accuracy than that of the recent, very important work on residual networks [3]. Furthermore, significantly improving the speed of high-quality detection can broaden the range of settings where computer vision is useful.

We summarize our contributions as follows:

• We introduce SSD, a single-shot detector for multiple categories that is faster than the previous state-of-the-art for single shot detectors (YOLO), and significantly more accurate, in fact as accurate as slower techniques that perform explicit region proposals and pooling (including Faster R-CNN).
• The core of SSD is predicting category scores and box offsets for a fixed set of default bounding boxes using small convolutional filters applied to feature maps.
• To achieve high detection accuracy we produce predictions of different scales from feature maps of different scales, and explicitly separate predictions by aspect ratio.
• These design features lead to simple end-to-end training and high accuracy, even on low resolution input images, further improving the speed vs accuracy trade-off.
• Experiments include timing and accuracy analysis on models with varying input size evaluated on PASCAL VOC, COCO, and ILSVRC and are compared to a range of recent state-of-the-art approaches.

We summarize our contributions as follows:

• We introduce SSD, a single-shot detector for multiple categories that is faster than the previous technique for single-shot detectors (YOLO), and it is much more accurate, as accurate as slower techniques that perform explicit region proposals and pooling (including Fast R-CNN).
• The core of SSD is predicting category scores and box offsets for a fixed set of default bounding boxes using small convolutional filters applied to feature maps.
• To achieve high detection accuracy, we produce predictions of different scales from feature maps of different scales, and explicitly separate predictions by aspect ratio.
• These design features provide simple end-to-end training and high accuracy even on low-resolution input images, further improving the speed vs. accuracy trade-off.
• Experiments include timing and accuracy analysis of models with various input sizes evaluated on PASCAL VOC, COCO, and ILSVRC, and are compared to a range of recent state-of-the-art approaches.

Fig. 1: SSD framework. (a) SSD only needs an input image and ground truth boxes for each object during training. In a convolutional fashion, we evaluate a small set (e.g. 4) of default boxes of different aspect ratios at each location in several feature maps with different scales (e.g. 8 × 8 and 4 × 4 in (b) and (c)). For each default box, we predict both the shape offsets and the confidences for all object categories ((c1, c2, ... , cp)). At training time, we first match these default boxes to the ground truth boxes. For example, we have matched two default boxes with the cat and one with the dog, which are treated as positives and the rest as negatives. The model loss is a weighted sum between localization loss (e.g. Smooth L1 [6]) and confidence loss (e.g. Softmax).

Fig. 1 : SSD framework. (a) SSD needs only an input image and ground truth boxes for each object during training. In a convolutional manner, we evaluate a small set (e.g. 4) of different aspect ratios at each location in several feature maps with different scales (e.g. 8x8 and 4x4 in (b) and (c)). For each default box, we predict the shape offsets and the confidences for all object categories ((c1, c2, ..., cp)). At training time, we first match these default boxes to the ground truth boxes. For example, we matched two default boxes with the cat, treated one as positive, and treated the rest as negatives. The model loss is a weighted sum between the localization loss (e.g. Smooth L1 [6]) and the confidence loss (e.g. Softmax).

The SSD approach is based on a feed-forward convolutional network that produces a fixed-size collection of bounding boxes and scores for the presence of object class instances in those boxes, followed by a non-maximum suppression step to produce the final detections. The early network layers are based on a standard architecture used for high quality image classification (truncated before any classification layers), which we will call the base network2. We then add auxiliary structure to the network to produce detections with the following key features:

The SSD approach is based on a feed-forward convolutional network that produces a fixed-size collection of bounding boxes and scores for the presence of object class instances in those boxes, followed by a non-maximum suppression step that produces the final detections. The early network layers are based on a standard architecture used for high-quality image classification (truncated before the classification layers), which we call the base network 2. We then add auxiliary structure to the network to produce detections with the following key features.

Multi-scale feature maps for detection We add convolutional feature layers to the end of the truncated base network. These layers decrease in size progressively and allow predictions of detections at multiple scales. The convolutional model for predicting detections is different for each feature layer (cf Overfeat[4] and YOLO[5] that operate on a single scale feature map).

Multi-scale feature maps for detection: We add convolutional feature layers to the end of the truncated base network. These layers decrease in size progressively and allow detections to be predicted at multiple scales. The convolutional model for predicting detections differs for each feature layer (cf Overfeat [4] and YOLO [5], which operate on a single-scale feature map).

Convolutional predictors for detection Each added feature layer (or optionally an existing feature layer from the base network) can produce a fixed set of detection predictions using a set of convolutional filters. These are indicated on top of the SSD network architecture in Fig. 2. For a feature layer of size m × n with p channels, the basic element for predicting parameters of a potential detection is a 3 × 3 × p small kernel that produces either a score for a category, or a shape offset relative to the default box coordinates. At each of the m × n locations where the kernel is applied, it produces an output value. The bounding box offset output values are measured relative to a default box position relative to each feature map location (cf the architecture of YOLO[5] that uses an intermediate fully connected layer instead of a convolutional filter for this step).

Convolutional predictors for detection: Each added feature layer (or optionally an existing feature layer from the base network) can produce a fixed set of detection predictions using a set of convolutional filters. These are shown at the top of the SSD network architecture in Fig. 1. 2. For a feature layer of size m × n with p channels, the basic element for predicting the parameters of a potential detection is a small 3x3xp kernel that produces a score for a category, or the default box coordinates. At each of the mxn locations where the kernel is applied, it produces an output value. The bounding box offset output values are measured relative to the default box position associated with each feature map location (cf the YOLO [5] architecture, which uses an intermediate fully connected layer instead of a convolutional filter for this step).

Default boxes and aspect ratios We associate a set of default bounding boxes with each feature map cell, for multiple feature maps at the top of the network. The default boxes tile the feature map in a convolutional manner, so that the position of each box relative to its corresponding cell is fixed. At each feature map cell, we predict the offsets relative to the default box shapes in the cell, as well as the per-class scores that indicate the presence of a class instance in each of those boxes. Specifically, for each box out of k at a given location, we compute c class scores and the 4 offsets relative to the original default box shape. This results in a total of (c + 4)k filters that are applied around each location in the feature map, yielding (c + 4)kmn outputs for a m × n feature map. For an illustration of default boxes, please refer to Fig. 1. Our default boxes are similar to the anchor boxes used in Faster R-CNN [2], however we apply them to several feature maps of different resolutions. Allowing different default box shapes in several feature maps let us efficiently discretize the space of possible output box shapes.

Default boxes and aspect ratios: We associate a set of default bounding boxes with each feature map cell, for multiple feature maps at the top of the network. The default boxes tile the feature map in a convolutional manner, so that the position of each box relative to its corresponding cell is fixed. At each feature map cell, we predict the offsets relative to the default box shapes in the cell, as well as per-class scores that indicate the presence of a class instance in each box. Specifically, for each of the k boxes at a given location, we compute c class scores and 4 offsets relative to the original default box shape. As a result, (c + 4) k filters are applied around each location in the feature map, yielding (c + 4) kmn outputs for an m × n feature map. For an illustration of default boxes, see Fig. 2. 1. Our default boxes are similar to the anchor boxes used in Faster R-CNN [2], but we apply them to several feature maps of different resolutions. Allowing different default box shapes in several feature maps lets us efficiently discretize the space of possible output box shapes.

Fig. 2: A comparison between two single shot detection models: SSD and YOLO [5]. Our SSD model adds several feature layers to the end of a base network, which predict the offsets to default boxes of different scales and aspect ratios and their associated confidences. SSD with a 300 × 300 input size significantly outperforms its 448 × 448 YOLO counterpart in accuracy on VOC2007 test while also improving the speed.

Fig. 2 : A comparison of two single-shot detection models, SSD and YOLO [5]. Our SSD model adds several feature layers to the end of the base network. These feature layers predict the offsets to default boxes of different scales and aspect ratios, and their associated confidences. SSD with a 300 × 300 input size far outperforms 448 × 448 YOLO in accuracy on the VOC2007 test while also improving speed.

The key difference between training SSD and training a typical detector that uses region proposals, is that ground truth information needs to be assigned to specific outputs in the fixed set of detector outputs. Some version of this is also required for training in YOLO[5] and for the region proposal stage of Faster R-CNN[2] and MultiBox[7]. Once this assignment is determined, the loss function and back propagation are applied endto-end. Training also involves choosing the set of default boxes and scales for detection as well as the hard negative mining and data augmentation strategies.

The key difference between training SSD and training a typical detector that uses region proposals is that ground truth information must be assigned to specific outputs in the fixed set of detector outputs. Some version of this is also required for training in YOLO [5] and for the region proposal stage of Faster R-CNN [2] and MultiBox [7]. Once this assignment is determined, the loss function and backpropagation are applied end-to-end. Training also involves choosing the set of default boxes and scales for detection, as well as hard negative mining and data augmentation strategies.

Matching strategy During training we need to determine which default boxes correspond to a ground truth detection and train the network accordingly. For each ground truth box we are selecting from default boxes that vary over location, aspect ratio, and scale. We begin by matching each ground truth box to the default box with the best jaccard overlap (as in MultiBox [7]). Unlike MultiBox, we then match default boxes to any ground truth with jaccard overlap higher than a threshold (0.5). This simplifies the learning problem, allowing the network to predict high scores for multiple overlapping default boxes rather than requiring it to pick only the one with maximum overlap.

Matching strategy: During training, we need to determine which default boxes correspond to a ground truth detection and train the network accordingly. For each ground truth box, we select from default boxes that vary by location, aspect ratio, and scale. We begin by matching each ground truth box to the default box with the best jaccard overlap (MultiBox [7]). Unlike MultiBox, we then match default boxes to ground truth with jaccard overlap higher than a threshold (0.5). This simplifies the learning problem, letting the network predict high scores for multiple overlapping default boxes instead of requiring it to pick only the one with maximum overlap.

Training objective The SSD training objective is derived from the MultiBox objective [7,8] but is extended to handle multiple object categories. Let \(x_{ij}^{p} = {1,0}\) be an indicator for matching the \(i\)-th default box to the \(j\)-th ground truth box of category \(p\). In the matching strategy above, we can have \(\sum_{i}x_{ij}^{p} \geq 1\). The overall objective loss function is a weighted sum of the localization loss (loc) and the confidence loss (conf):

Training objective: The SSD training objective is derived from the MultiBox objective [7,8] but is extended to handle multiple object categories. Let \(x_{ij}^{p} = {1,0}\) be an indicator for matching the i-th default box to the j-th ground truth box of category p. In the matching strategy above, we can have \(\sum_{i}x_{ij}^{p} \geq 1\). The overall objective loss function is a weighted sum of the localization loss (loc) and the confidence loss (conf).

where N is the number of matched default boxes. If N = 0, wet set the loss to 0. The localization loss is a Smooth L1 loss [6] between the predicted box (l) and the ground truth box (g) parameters. Similar to Faster R-CNN [2], we regress to offsets for the center (cx, cy) of the default bounding box (d) and for its width (w) and height (h).

Here N is the number of matched default boxes. If N = 0, we set the loss to 0. The localization loss is a Smooth L1 loss between the predicted box (l) and the ground truth box (g) parameters [6]. As in Faster R-CNN [2], we regress to offsets for the center (cx, cy) of the default bounding box (d) and for its width (w) and height (h).

The confidence loss is the softmax loss over multiple classes confidences (c).

The confidence loss is the softmax loss over multiple class confidences (c).

and the weight term α is set to 1 by cross validation.

The weight term α is set to 1 by cross validation.

Choosing scales and aspect ratios for default boxes To handle different object scales, some methods [4,9] suggest processing the image at different sizes and combining the results afterwards. However, by utilizing feature maps from several different layers in a single network for prediction we can mimic the same effect, while also sharing parameters across all object scales. Previous works [10,11] have shown that using feature maps from the lower layers can improve semantic segmentation quality because the lower layers capture more fine details of the input objects. Similarly, [12] showed that adding global context pooled from a feature map can help smooth the segmentation results. Motivated by these methods, we use both the lower and upper feature maps for detection. Figure 1 shows two exemplar feature maps (8×8 and 4×4) which are used in the framework. In practice, we can use many more with small computational overhead.

Choosing scales and aspect ratios for default boxes: To handle different object sizes, some methods [4,9] suggest processing the image at different sizes and combining the results afterwards. However, by using feature maps from several layers in a single network for prediction, we can mimic the same effect while sharing parameters across all object scales. Previous studies [10,11] have shown that using feature maps from lower layers can improve semantic segmentation quality, because the lower layers capture finer details of the input objects. Similarly, [12] showed that adding global context pooled from a feature map can help smooth the segmentation results. Motivated by these methods, we use both lower and upper feature maps for detection. Figure 1 shows two exemplar feature maps (8 × 8 and 4 × 4) used in the framework. In practice, we can use many more with small computational overhead.

Feature maps from different levels within a network are known to have different (empirical) receptive field sizes [13]. Fortunately, within the SSD framework, the default boxes do not necessary need to correspond to the actual receptive fields of each layer. We design the tiling of default boxes so that specific feature maps learn to be responsive to particular scales of the objects. Suppose we want to use m feature maps for prediction. The scale of the default boxes for each feature map is computed as:

Feature maps from different levels within a network are known to have different (empirical) receptive field sizes [13]. Fortunately, within the SSD framework, the default boxes do not need to correspond to the actual receptive fields of each layer. We design the tiling of default boxes so that specific feature maps learn to respond to particular scales of objects. Suppose we use m feature maps for prediction. The scale of the default boxes for each feature map is computed as follows.

where \(s_{min}\) is 0.2 and \(s_{max}\) is 0.9, meaning the lowest layer has a scale of 0.2 and the highest layer has a scale of 0.9, and all layers in between are regularly spaced. We impose different aspect ratios for the default boxes, and denote them as \(a_r \in {1,2,3,\frac{1}{2},\frac{1}{3}}\). We can compute the width \((w_{k}^{a} = s_k \sqrt{a_r}\) and height \((h_k^a = s_k / \sqrt{a_r})\) for each default box. For the aspect ratio of 1, we also add a default box whose scale is \(s'_k = \sqrt{s_{k}s_{k+1}}\) resulting in 6 default boxes per feature map location. We set the center of each default box to \((\frac{i+0.5}{|f_k|}, \frac{j+0.5}{|f_k|})\) , where \(|f_k|\) is the size of the \(k\)-th square feature map, \(i, j \in [0, |fk|)\). In practice, one can also design a distribution of default boxes to best fit a specific dataset. How to design the optimal tiling is an open question as well.

Here \(s_{min}\) is 0.2 and \(s_{max}\) is 0.9, so the lowest layer has a scale of 0.2, the highest layer has a scale of 0.9, and all layers in between are regularly spaced. Different aspect ratios are imposed on the default boxes, denoted \(a_r \in {1,2,3,\frac{1}{2},\frac{1}{3}}\). We can compute the width \((w_{k}^{a} = s_k \sqrt{a_r}\) and height \((h_k^a = s_k / \sqrt{a_r})\) of each default box. For an aspect ratio of 1, we add a default box whose scale is \(s'_k = \sqrt{s_{k}s_{k+1}}\), producing 6 default boxes per feature map location. We set the center of each default box to \((\frac{i+0.5}{|f_k|}, \frac{j+0.5}{|f_k|})\), where \(|f_k|\) is the size of the k-th square feature map, \(i, j \in [0, |fk|)\). In practice, one can also design default boxes to best fit a specific dataset. How to design the optimal tiling is also an open question.

By combining predictions for all default boxes with different scales and aspect ratios from all locations of many feature maps, we have a diverse set of predictions, covering various input object sizes and shapes. For example, in Fig. 1, the dog is matched to a default box in the 4 × 4 feature map, but not to any default boxes in the 8 × 8 feature map. This is because those boxes have different scales and do not match the dog box, and therefore are considered as negatives during training.

By combining predictions for all default boxes with different scales and aspect ratios from all locations of many feature maps, we get a diverse set of predictions covering various input object sizes and shapes. For example, in Fig. 1, the dog is matched to a default box in the 4x4 feature map but not to any default box in the 8x8 feature map. This is because those boxes have different scales and do not match the dog box, so they are considered negatives during training.

Hard negative mining After the matching step, most of the default boxes are negatives, especially when the number of possible default boxes is large. This introduces a significant imbalance between the positive and negative training examples. Instead of using all the negative examples, we sort them using the highest confidence loss for each default box and pick the top ones so that the ratio between the negatives and positives is at most 3:1. We found that this leads to faster optimization and a more stable training.

Hard negative mining: After the matching step, most of the default boxes are negatives, especially when the number of possible default boxes is large. This introduces a significant imbalance between the positive and negative training examples. Instead of using all the negative examples, we sort them by the highest confidence loss for each default box and pick the top ones so that the ratio of negatives to positives is at most 3 : 1. We found that this leads to faster optimization and more stable training.

Data augmentation To make the model more robust to various input object sizes and shapes, each training image is randomly sampled by one of the following options:

• Use the entire original input image.
• Sample a patch so that the minimum jaccard overlap with the objects is 0.1, 0.3, 0.5, 0.7, or 0.9.
• Randomly sample a patch.

Data augmentation: To make the model more robust to various input object sizes and shapes, each training image is randomly sampled by one of the following options.

• Use the entire original input image.
• Sample a patch so that the minimum jaccard is 0.1, 0.3, 0.5, 0.7, or 0.9.
• Randomly sample a patch.

The size of each sampled patch is [0.1, 1] of the original image size, and the aspect ratio is between 1/2 and 2. We keep the overlapped part of the ground truth box if the center of it is in the sampled patch. After the aforementioned sampling step, each sampled patch is resized to fixed size and is horizontally flipped with probability of 0.5, in addition to applying some photo-metric distortions similar to those described in [14].

The size of each sampled patch is [0.1, 1] of the original image size, and the aspect ratio is between 1/2 and 2. We keep the overlapped part of the ground truth box if its center is in the sampled patch . After the aforementioned sampling step, each sampled patch is resized to a fixed size and horizontally flipped with a probability of 0.5, in addition to applying some photometric distortions similar to those described in [14].