Faster-RCNN:Paper

State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet ⁴⁰ and Fast R-CNN ⁴¹ have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with “attention” mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model ⁴², our detection system has a frame rate of 5fps (including all steps) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available.

State-of-the-art object detection networks depend on region proposal algorithms to hypothesize the locations of objects. Advances like SPPnet ⁴³ and Fast R-CNN ⁴⁴ are reducing the running time of detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We also merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with "attention" mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model ⁴⁵, our detection system has a frame rate of 5fps (including all steps) on a GPU, yet achieves state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the first-place entries winning several tracks. The code has been made publicly available.

Recent advances in object detection are driven by the success of region proposal methods (e.g., ⁴⁶) and region-based convolutional neural networks (RCNNs) ⁴⁷. Although region-based CNNs were computationally expensive as originally developed in ⁴⁸, their cost has been drastically reduced thanks to sharing convolutions across proposals ⁴⁹, ⁵⁰. The latest incarnation, Fast R-CNN ⁵¹, achieves near real-time rates using very deep networks ⁵², when ignoring the time spent on region proposals. Now, proposals are the test-time computational bottleneck in state-of-the-art detection systems.

Recent advances in object detection are driven by the success of region proposal methods (e.g., ⁵³) and region-based convolutional neural networks (RCNNs) ⁵⁴. Region-based CNNs were computationally expensive as originally developed in ⁵⁵, but their cost has been drastically reduced thanks to sharing convolutions across proposals ⁵⁶, ⁵⁷. The latest incarnation, Fast R-CNN ⁵⁸, achieves near real-time rates using very deep networks ⁵⁹, when ignoring the time spent on region proposals. Now, proposals are the test-time computational bottleneck of state-of-the-art detection systems.

Region proposal methods typically rely on inexpensive features and economical inference schemes. Selective Search ⁶⁰, one of the most popular methods, greedily merges superpixels based on engineered low-level features. Yet when compared to efficient detection networks ⁶¹, Selective Search is an order of magnitude slower, at 2 seconds per image in a CPU implementation. EdgeBoxes ⁶² currently provides the best tradeoff between proposal quality and speed, at 0.2 seconds per image. Nevertheless, the region proposal step still consumes as much running time as the detection network.

Region proposal methods typically rely on inexpensive features and economical inference schemes. Selective Search ⁶³, one of the most popular methods, greedily merges superpixels based on engineered low-level features. Yet compared to efficient detection networks ⁶⁴, Selective Search is an order of magnitude slower, at 2 seconds per image in a CPU implementation. EdgeBoxes ⁶⁵ currently provides the best tradeoff between proposal quality and speed, at 0.2 seconds per image. Nevertheless, the region proposal step still consumes as much running time as the detection network.

One may note that fast region-based CNNs take advantage of GPUs, while the region proposal methods used in research are implemented on the CPU, making such runtime comparisons inequitable. An obvious way to accelerate proposal computation is to reimplement it for the GPU. This may be an effective engineering solution, but re-implementation ignores the down-stream detection network and therefore misses important opportunities for sharing computation.

Fast region-based CNNs can take advantage of GPUs, while the region proposal methods used in research are implemented on the CPU, making such runtime comparisons unfair. An obvious way to accelerate proposal computation is to reimplement it for the GPU. This may be an effective engineering solution, but reimplementation ignores the downstream detection network and therefore misses important opportunities for sharing computation.

In this paper, we show that an algorithmic change—computing proposals with a deep convolutional neural network—leads to an elegant and effective solution where proposal computation is nearly cost-free given the detection network’s computation. To this end, we introduce novel Region Proposal Networks (RPNs) that share convolutional layers with state-of-the-art object detection networks ⁶⁶, ⁶⁷. By sharing convolutions at test-time, the marginal cost for computing proposals is small (e.g., 10ms per image).

In this paper, we show that an algorithmic change—computing proposals with a deep convolutional neural network—leads to an elegant and efficient solution where proposal computation is nearly cost-free given the detection network's computation. To this end, we introduce novel Region Proposal Networks (RPNs) that share convolutional layers with state-of-the-art object detection networks ⁶⁸, ⁶⁹. By sharing convolutions at test time, the marginal cost for computing proposals is small (e.g., 10ms per image).

Our observation is that the convolutional feature maps used by region-based detectors, like Fast RCNN, can also be used for generating region proposals. On top of these convolutional features, we construct an RPN by adding a few additional convolutional layers that simultaneously regress region bounds and objectness scores at each location on a regular grid. The RPN is thus a kind of fully convolutional network (FCN) ⁷⁰ and can be trained end-toend specifically for the task for generating detection proposals.

Our observation is that the convolutional feature maps used by region-based detectors, like Fast RCNN, can also be used for generating region proposals. On top of these convolutional features, we construct an RPN by adding a few additional convolutional layers that simultaneously regress region bounds and objectness scores at each location on a regular grid. The RPN is thus a kind of fully convolutional network (FCN) ⁷¹ and can be trained end-toend specifically for the task of generating detection proposals.

RPNs are designed to efficiently predict region proposals with a wide range of scales and aspect ratios. In contrast to prevalent methods ⁷², ⁷³, ⁷⁴, ⁷⁵ that use pyramids of images (Figure 1, a) or pyramids of filters (Figure 1, b), we introduce novel “anchor” boxes that serve as references at multiple scales and aspect ratios. Our scheme can be thought of as a pyramid of regression references (Figure 1, c), which avoids enumerating images or filters of multiple scales or aspect ratios. This model performs well when trained and tested using single-scale images and thus benefits running speed.

RPNs are designed to efficiently predict region proposals with a wide range of scales and aspect ratios. Unlike prevalent methods ⁷⁶, ⁷⁷, ⁷⁸, ⁷⁹ that use pyramids of images (Figure 1) or pyramids of filters (Figure 1, b), we introduce novel "anchor" boxes that serve as references at multiple scales and aspect ratios. Our scheme can be regarded as a pyramid of regression references (Figure 1 c), which avoids enumerating images or filters of multiple scales or aspect ratios. This model performs well when trained and tested using single-scale images, and thus benefits running speed.

Figure 1: Different schemes for addressing multiple scales and sizes. (a) Pyramids of images and feature maps are built, and the classifier is run at all scales. (b) Pyramids of filters with multiple scales/sizes are run on the feature map. (c) We use pyramids of reference boxes in the regression functions.

To unify RPNs with Fast R-CNN ⁸⁰ object detection networks, we propose a training scheme that alternates between fine-tuning for the region proposal task and then fine-tuning for object detection, while keeping the proposals fixed. This scheme converges quickly and produces a unified network with convolutional features that are shared between both tasks. (Since the publication of the conference version of this paper ⁸¹, we have also found that RPNs can be trained jointly with Fast R-CNN networks leading to less training time)

To unify RPNs with Fast R-CNN ⁸² object detection networks, we propose a training scheme that alternates between fine-tuning for the region proposal task and then fine-tuning for object detection, while keeping the proposals fixed. This scheme converges quickly and produces a unified network with convolutional features shared between both tasks. (Since the publication of the conference version of this paper ⁸³, we have also found that RPNs can be trained jointly with Fast R-CNN networks, leading to less training time)

We comprehensively evaluate our method on the PASCAL VOC detection benchmarks ⁸⁴ where RPNs with Fast R-CNNs produce detection accuracy better than the strong baseline of Selective Search with Fast R-CNNs. Meanwhile, our method waives nearly all computational burdens of Selective Search at test-time—the effective running time for proposals is just 10 milliseconds. Using the expensive very deep models of ⁸⁵, our detection method still has a frame rate of 5fps (including all steps) on a GPU, and thus is a practical object detection system in terms of both speed and accuracy. We also report results on the MS COCO dataset ⁸⁶ and investigate the improvements on PASCAL VOC using the COCO data. Code has been made publicly available at https://github.com/shaoqingren/faster_rcnn (in MATLAB) and https://github.com/rbgirshick/py-faster-rcnn (in Python).

We comprehensively evaluate our method on the PASCAL VOC detection benchmarks ⁸⁷, where RPNs with Fast R-CNNs produce detection accuracy better than the strong baseline of Selective Search with Fast R-CNNs. Meanwhile, our method waives nearly all computational burdens of Selective Search at test time—the effective running time for proposals is 10 milliseconds. Using the expensive very deep models of ⁸⁸, our detection method still has a frame rate of 5fps (including all steps) on a GPU, and thus is a practical object detection system in terms of both speed and accuracy. We also report results on the MS COCO dataset ⁸⁹ and investigate the improvements on PASCAL VOC using the COCO data. The code has been made publicly available at https://github.com/shaoqingren/faster_rcnn (in MATLAB) and https://github.com/rbgirshick/py-faster-rcnn (in Python).

A preliminary version of this manuscript was published previously ⁹⁰. Since then, the frameworks of RPN and Faster R-CNN have been adopted and generalized to other methods, such as 3D object detection ⁹¹, part-based detection ⁹², instance segmentation ⁹³, and image captioning ⁹⁴. Our fast and effective object detection system has also been built in commercial systems such as at Pinterests ⁹⁵, with user engagement improvements reported.

A preliminary version of this manuscript was published previously ⁹⁶. Since then, the frameworks of RPN and Faster R-CNN have been adopted and generalized to other methods, such as 3D object detection ⁹⁷, part-based detection ⁹⁸, instance segmentation ⁹⁹, and image captioning ¹⁰⁰. Our fast and effective object detection system has also been built into commercial systems such as at Pinterests ¹⁰¹, with reported improvements in user engagement.

In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the basis of several 1st-place entries ¹⁰² in the tracks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. RPNs completely learn to propose regions from data, and thus can easily benefit from deeper and more expressive features (such as the 101-layer residual nets adopted in ¹⁰³). Faster R-CNN and RPN are also used by several other leading entries in these competitions ( http://image-net.org/challenges/LSVRC/2015/results ). These results suggest that our method is not only a cost-efficient solution for practical usage, but also an effective way of improving object detection accuracy

In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the basis of several 1st-place entries ¹⁰⁴ in the tracks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. RPNs completely learn to propose regions from data, and thus can easily benefit from deeper and more expressive features (such as the 101-layer residual nets adopted in ¹⁰⁵). Faster R-CNN and RPN are also used by several other leading entries in these competitions ( http://image-net.org/challenges/LSVRC/2015/results ). These results suggest that our method is not only a cost-efficient solution for practical usage, but also an effective way of improving object detection accuracy

Object Proposals. There is a large literature on object proposal methods. Comprehensive surveys and comparisons of object proposal methods can be found in ¹⁰⁶, ¹⁰⁷, ¹⁰⁸. Widely used object proposal methods include those based on grouping super-pixels (e.g., Selective Search ¹⁰⁹, CPMC ¹¹⁰, MCG ¹¹¹) and those based on sliding windows (e.g., objectness in windows ¹¹², EdgeBoxes ¹¹³). Object proposal methods were adopted as external modules independent of the detectors (e.g., Selective Search ¹¹⁴ object detectors, RCNN ¹¹⁵, and Fast R-CNN ¹¹⁶).

Object Proposals. There is a large literature on object proposal methods. Comprehensive surveys and comparisons of object proposal methods can be found in ¹¹⁷, ¹¹⁸, ¹¹⁹. Widely used object proposal methods include those based on grouping superpixels (e.g., Selective Search ¹²⁰, CPMC ¹²¹, MCG ¹²²) and those based on sliding windows (e.g., objectness in windows ¹²³, EdgeBoxes ¹²⁴). Object proposal methods were adopted as external modules independent of the detectors (e.g., Selective Search ¹²⁵ object detectors, RCNN ¹²⁶, and Fast R-CNN ¹²⁷).

Deep Networks for Object Detection. The R-CNN method ¹²⁸ trains CNNs end-to-end to classify the proposal regions into object categories or background. R-CNN mainly plays as a classifier, and it does not predict object bounds (except for refining by bounding box regression). Its accuracy depends on the performance of the region proposal module (see comparisons in ¹²⁹). Several papers have proposed ways of using deep networks for predicting object bounding boxes ¹³⁰, ¹³¹, ¹³², ¹³³. In the OverFeat method ¹³⁴, a fully-connected layer is trained to predict the box coordinates for the localization task that assumes a single object. The fully-connected layer is then turned into a convolutional layer for detecting multiple classspecific objects. The MultiBox methods ¹³⁵, ¹³⁶ generate region proposals from a network whose last fully-connected layer simultaneously predicts multiple class-agnostic boxes, generalizing the “singlebox” fashion of OverFeat. These class-agnostic boxes are used as proposals for R-CNN ¹³⁷. The MultiBox proposal network is applied on a single image crop or multiple large image crops (e.g., 224×224), in contrast to our fully convolutional scheme. MultiBox does not share features between the proposal and detection networks. We discuss OverFeat and MultiBox in more depth later in context with our method. Concurrent with our work, the DeepMask method ¹³⁸ is developed for learning segmentation proposals.

Deep Networks for Object Detection. The R-CNN method ¹³⁹ trains CNNs end-to-end to classify proposal regions into object categories or background. R-CNN mainly plays the role of a classifier, and it does not predict object bounds (except for refining by bounding box regression). Its accuracy depends on the performance of the region proposal module (see comparisons in ¹⁴⁰). Several papers have proposed ways of using deep networks to predict object bounding boxes ¹⁴¹, ¹⁴², ¹⁴³, ¹⁴⁴. In the OverFeat method ¹⁴⁵, a fully-connected layer is trained to predict the box coordinates for the localization task that assumes a single object. The fully-connected layer is then turned into a convolutional layer for detecting multiple classspecific objects. The MultiBox methods ¹⁴⁶, ¹⁴⁷ generate region proposals from a network whose last fully-connected layer simultaneously predicts multiple class-agnostic boxes, generalizing the singlebox fashion of OverFeat. These class-agnostic boxes are used as proposals for R-CNN ¹⁴⁸. The MultiBox proposal network is applied to a single image crop or multiple large image crops (e.g., 224 × 224), in contrast to our fully convolutional scheme. MultiBox does not share features between the proposal and detection networks. We discuss OverFeat and MultiBox in more depth later in relation to our method. Concurrent with our work, the DeepMask method ¹⁴⁹ is being developed for learning segmentation proposals.

Shared computation of convolutions ¹⁵⁰, ¹⁵¹, ¹⁵², ¹⁵³, ¹⁵⁴ has been attracting increasing attention for efficient, yet accurate, visual recognition. The OverFeat paper ¹⁵⁵ computes convolutional features from an image pyramid for classification, localization, and detection. Adaptively-sized pooling (SPP) ¹⁵⁶ on shared convolutional feature maps is developed for efficient region-based object detection ¹⁵⁷, ¹⁵⁸ and semantic segmentation ¹⁵⁹. Fast R-CNN ¹⁶⁰ enables end-to-end detector training on shared convolutional features and shows compelling accuracy and speed.

Shared computation of convolutions ¹⁶¹, ¹⁶², ¹⁶³, ¹⁶⁴, ¹⁶⁵ has been attracting attention for efficient yet accurate visual recognition. The OverFeat paper ¹⁶⁶ computes convolutional features from an image pyramid for classification, localization, and detection. Adaptively-sized pooling (SPP) ¹⁶⁷ on shared convolutional feature maps is being developed for efficient region-based object detection ¹⁶⁸, ¹⁶⁹ and semantic segmentation ¹⁷⁰. Fast R-CNN ¹⁷¹ enables end-to-end detector training on shared convolutional features and shows excellent accuracy and speed.

Our object detection system, called Faster R-CNN, is composed of two modules. The first module is a deep fully convolutional network that proposes regions, and the second module is the Fast R-CNN detector ¹⁷² that uses the proposed regions. The entire system is a single, unified network for object detection (Figure 2). Using the recently popular terminology of neural networks with ‘attention’ ¹⁷³ mechanisms, the RPN module tells the Fast R-CNN module where to look. In Section 3.1 we introduce the designs and properties of the network for region proposal. In Section 3.2 we develop algorithms for training both modules with features shared.

Our object detection system, called Faster R-CNN, is composed of two modules. The first module is a deep fully convolutional network that proposes regions, and the second module is the Fast R-CNN detector ¹⁷⁴ that uses the proposed regions. The entire system is a single, unified network for object detection (Figure 2). Using the recently popular terminology of neural networks with 'attention' ¹⁷⁵ mechanisms, the RPN module tells the Fast R-CNN module where to look. In Section 3.1 we introduce the design and properties of the network for region proposal. In Section 3.2 we develop algorithms for training both modules with shared features.

A Region Proposal Network (RPN) takes an image (of any size) as input and outputs a set of rectangular object proposals, each with an objectness score. (“Region” is a generic term and in this paper we only consider rectangular regions, as is common for many methods (e.g., ¹⁷⁶, ¹⁷⁷, ¹⁷⁸). “Objectness” measures membership to a set of object classes vs. background.) We model this process with a fully convolutional network ¹⁷⁹, which we describe in this section. Because our ultimate goal is to share computation with a Fast R-CNN object detection network ¹⁸⁰, we assume that both nets share a common set of convolutional layers. In our experiments, we investigate the Zeiler and Fergus model ¹⁸¹ (ZF), which has 5 shareable convolutional layers and the Simonyan and Zisserman model ¹⁸² (VGG-16), which has 13 shareable convolutional layers.

A Region Proposal Network (RPN) takes an image (of any size) as input and outputs a set of rectangular object proposals, each with an objectness score. ("Region" is a generic term, and in this paper we consider only rectangular regions, as is common for many methods (e.g., ¹⁸³, ¹⁸⁴, ¹⁸⁵). "Objectness" measures membership to a set of object classes vs. background.) We model this process with a fully convolutional network ¹⁸⁶, which we describe in this section. Because our ultimate goal is to share computation with a Fast R-CNN object detection network ¹⁸⁷, we assume that both nets share a common set of convolutional layers. In our experiments, we investigate the Zeiler and Fergus model ¹⁸⁸ (ZF), which has 5 shareable convolutional layers, and the Simonyan and Zisserman model ¹⁸⁹ (VGG-16), which has 13 shareable convolutional layers.

To generate region proposals, we slide a small network over the convolutional feature map output by the last shared convolutional layer. This small network takes as input an n × n spatial window of the input convolutional feature map. Each sliding window is mapped to a lower-dimensional feature (256-d for ZF and 512-d for VGG, with ReLU ¹⁹⁰ following). This feature is fed into two sibling fullyconnected layers—a box-regression layer (reg) and a box-classification layer (cls). We use n = 3 in this paper, noting that the effective receptive field on the input image is large (171 and 228 pixels for ZF and VGG, respectively). This mini-network is illustrated at a single position in Figure 3 (left). Note that because the mini-network operates in a sliding-window fashion, the fully-connected layers are shared across all spatial locations. This architecture is naturally implemented with an n×n convolutional layer followed by two sibling 1 × 1 convolutional layers (for reg and cls, respectively).

To generate region proposals, we slide a small network over the convolutional feature map output by the last shared convolutional layer. This small network takes as input an n × n spatial window of the input convolutional feature map. Each sliding window is mapped to a lower-dimensional feature (256-d for ZF and 512-d for VGG, with ReLU ¹⁹¹ following). This feature is fed into two sibling fullyconnected layers—a box-regression layer (reg) and a box-classification layer (cls). We use n = 3 in this paper, noting that the effective receptive field on the input image is large (171 and 228 pixels for ZF and VGG, respectively). This mini-network is illustrated at a single position in Figure 3 (left). Because the mini-network operates in a sliding-window fashion, the fully-connected layers are shared across all spatial locations. This architecture is naturally implemented with an n × n convolutional layer followed by two sibling 1 × 1 convolutional layers (for reg and cls, respectively).

Figure 3: Left: Region Proposal Network (RPN). Right: Example detections using RPN proposals on PASCAL VOC 2007 test. Our method detects objects in a wide range of scales and aspect ratios.

At each sliding-window location, we simultaneously predict multiple region proposals, where the number of maximum possible proposals for each location is denoted as k. So the reg layer has 4k outputs encoding the coordinates of k boxes, and the cls layer outputs 2k scores that estimate probability of object or not object for each proposal (For simplicity we implement the cls layer as a two-class softmax layer. Alternatively, one may use logistic regression to produce k scores.). The k proposals are parameterized relative to k reference boxes, which we call anchors. An anchor is centered at the sliding window in question, and is associated with a scale and aspect ratio (Figure 3, left). By default we use 3 scales and 3 aspect ratios, yielding k = 9 anchors at each sliding position. For a convolutional feature map of a size W × H (typically ∼2,400), there are W Hk anchors in total.

At each sliding-window location, we simultaneously predict multiple region proposals, where the number of maximum possible proposals for each location is denoted as k. So the reg layer has 4k outputs encoding the coordinates of k boxes, and the cls layer outputs 2k scores that estimate the probability of object or not object for each proposal,(For simplicity we implement the cls layer as a two-class softmax layer. Alternatively, one may use logistic regression to produce k scores.). The k proposals are parameterized relative to k reference boxes, which we call anchors. An anchor is centered at the sliding window in question, and is associated with a scale and aspect ratio (Figure 3, left). By default we use 3 scales and 3 aspect ratios, yielding k = 9 anchors at each sliding position. For a convolutional feature map of a size W × H (typically ~2,400), there are WHk anchors in total.

Translation-Invariant Anchors

Translation-Invariant Anchors

An important property of our approach is that it is translation invariant, both in terms of the anchors and the functions that compute proposals relative to the anchors. If one translates an object in an image, the proposal should translate and the same function should be able to predict the proposal in either location. This translation-invariant property is guaranteed by our method (As is the case of FCNs ¹⁹², our network is translation invariant up to the network’s total stride.). As a comparison, the MultiBox method ¹⁹³ uses k-means to generate 800 anchors, which are not translation invariant. So MultiBox does not guarantee that the same proposal is generated if an object is translated.

An important property of our approach is that it is translation invariant, both in terms of the anchors and the functions that compute proposals relative to the anchors. If one translates an object in an image, the proposal should translate and the same function should be able to predict the proposal in either location. This translation-invariant property is guaranteed by our method (As is the case of FCNs ¹⁹⁴, our network is translation invariant up to the network's total stride.). In comparison, the MultiBox method ¹⁹⁵ uses k-means to generate 800 anchors, which are not translation invariant. So MultiBox does not guarantee that the same proposal is generated if an object is translated.

The translation-invariant property also reduces the model size. MultiBox has a (4 + 1) × 800-dimensional fully-connected output layer, whereas our method has a (4 + 2) × 9-dimensional convolutional output layer in the case of k = 9 anchors. As a result, our output layer has 2.8 × 104 parameters (512 × (4 + 2) × 9 for VGG-16), two orders of magnitude fewer than MultiBox’s output layer that has 6.1 × 106 parameters (1536 × (4 + 1) × 800 for GoogleNet ¹⁹⁶ in MultiBox ¹⁹⁷). If considering the feature projection layers, our proposal layers still have an order of magnitude fewer parameters than MultiBox (Considering the feature projection layers, our proposal layers’ parameter count is 3 × 3 × 512 × 512 + 512 × 6 × 9 = 2.4 × 10^6; MultiBox’s proposal layers’ parameter count is 7 × 7 × (64 + 96 + 64 + 64) × 1536 + 1536 × 5 × 800 = 27 × 10^6.). We expect our method to have less risk of overfitting on small datasets, like PASCAL VOC.

The translation-invariant property also reduces the model size. Our method has a (4 + 2) × 9-dimensional convolutional output layer in the case of k = 9 anchors, whereas MultiBox has a (4 + 1) × 800-dimensional fully-connected output layer. As a result, our output layer has 2.8 × 104 parameters (512 × (4 + 2) × 9 for VGG-16), two orders of magnitude fewer than MultiBox's output layer, which has 6.1 × 106 parameters (1536 × (4 + 1) × 800 ¹⁹⁸ for GoogleNet ¹⁹⁹ in MultiBox). Considering the feature projection layers, our proposal layers still have an order of magnitude fewer parameters than MultiBox (Considering the feature projection layers, our proposal layers' parameter count is 3 × 3 × 512 × 512 + 512 × 6 × 9 = 2.4 × 10 ^ 6; MultiBox's proposal layers' parameter count is 7 × 7 × (64 + 96 + 64 + 64) × 1536 + 1536 × 5 × 800 = 27 × 10 ^ 6.). We expect our method to have less risk of overfitting on small datasets like PASCAL VOC.

Multi-Scale Anchors as Regression References

Multi-Scale Anchors as Regression References

Our design of anchors presents a novel scheme for addressing multiple scales (and aspect ratios). As shown in Figure 1, there have been two popular ways for multi-scale predictions. The first way is based on image/feature pyramids, e.g., in DPM ²⁰⁰ and CNNbased methods ²⁰¹, ²⁰², ²⁰³. The images are resized at multiple scales, and feature maps (HOG ²⁰⁴ or deep convolutional features ²⁰⁵, ²⁰⁶, ²⁰⁷) are computed for each scale (Figure 1(a)). This way is often useful but is time-consuming. The second way is to use sliding windows of multiple scales (and/or aspect ratios) on the feature maps. For example, in DPM ²⁰⁸, models of different aspect ratios are trained separately using different filter sizes (such as 5×7 and 7×5). If this way is used to address multiple scales, it can be thought of as a “pyramid of filters” (Figure 1(b)). The second way is usually adopted jointly with the first way ²⁰⁹.

Our design of anchors presents a new scheme for addressing multiple scales (and aspect ratios). As shown in Figure 1, there have been two popular ways for multi-scale prediction. The first way is based on image/feature pyramids, e.g., in DPM ²¹⁰ and CNNbased methods ²¹¹, ²¹², ²¹³. The images are resized at multiple scales, and feature maps (HOG ²¹⁴ or deep convolutional features ²¹⁵, ²¹⁶, ²¹⁷) are computed for each scale (Figure 1 (a)). This way is useful but time-consuming. The second way is to use sliding windows of multiple scales (and/or aspect ratios) on the feature maps. For example, in DPM ²¹⁸, models of different aspect ratios are trained separately using filters of different sizes (such as 5 × 7 and 7 × 5). If this way is used to address multiple scales, it can be thought of as a "pyramid of filters" (Figure 1 (b)). The second way is usually adopted jointly with the first way ²¹⁹.

As a comparison, our anchor-based method is built on a pyramid of anchors, which is more cost-efficient. Our method classifies and regresses bounding boxes with reference to anchor boxes of multiple scales and aspect ratios. It only relies on images and feature maps of a single scale, and uses filters (sliding windows on the feature map) of a single size. We show by experiments the effects of this scheme for addressing multiple scales and sizes (Table 8).

In comparison, our anchor-based method is built on a pyramid of anchors, which is more cost-efficient. Our method classifies and regresses bounding boxes with reference to anchor boxes of multiple scales and aspect ratios. It relies only on images and feature maps of a single scale, and uses filters (sliding windows on the feature map) of a single size. We show by experiments the effects of this scheme for addressing multiple scales and sizes (Table 8).

Because of this multi-scale design based on anchors, we can simply use the convolutional features computed on a single-scale image, as is also done by the Fast R-CNN detector ²²⁰. The design of multiscale anchors is a key component for sharing features without extra cost for addressing scales.

Because of this multi-scale design based on anchors, we can simply use the convolutional features computed on a single-scale image, as is also done by the Fast R-CNN detector ²²¹. The design of multi-scale anchors is a key component for sharing features without extra cost for addressing scales.

For training RPNs, we assign a binary class label (of being an object or not) to each anchor. We assign a positive label to two kinds of anchors: (i) the anchor/anchors with the highest Intersection-overUnion (IoU) overlap with a ground-truth box, or (ii) an anchor that has an IoU overlap higher than 0.7 with any ground-truth box. Note that a single ground-truth box may assign positive labels to multiple anchors. Usually the second condition is sufficient to determine the positive samples; but we still adopt the first condition for the reason that in some rare cases the second condition may find no positive sample. We assign a negative label to a non-positive anchor if its IoU ratio is lower than 0.3 for all ground-truth boxes. Anchors that are neither positive nor negative do not contribute to the training objective.

For training RPNs, we assign a binary class label (of being an object or not) to each anchor. We assign a positive label to two kinds of anchors: (i) the anchor/anchors with the highest Intersection-over-Union (IoU) overlap with a ground-truth box, or (ii) an anchor that has an IoU overlap higher than 0.7 with any ground-truth box. A single ground-truth box may assign positive labels to multiple anchors. Usually the second condition is sufficient to determine the positive samples; but we still adopt the first condition because in some rare cases the second condition may find no positive sample. We assign a negative label to a non-positive anchor if its IoU ratio is lower than 0.3 for all ground-truth boxes. Anchors that are neither positive nor negative do not contribute to the training objective.

With these definitions, we minimize an objective function following the multi-task loss in Fast R-CNN ²²². Our loss function for an image is defined as:

With these definitions, we can minimize an objective function following the multi-task loss in Fast R-CNN ²²³. Our loss function for an image is defined as follows:

Here, i is the index of an anchor in a mini-batch and p_i is the predicted probability of anchor i being an object. The ground-truth label p_i^* is 1 if the anchor is positive, and is 0 if the anchor is negative. t_i is a vector representing the 4 parameterized coordinates of the predicted bounding box, and t_i^* is that of the ground-truth box associated with a positive anchor. The classification loss Lcls is log loss over two classes (object vs. not object). For the regression loss, we use Lreg (t_i, t_i^) = R(t_i − t_i^) where R is the robust loss function (smooth L1) defined in ²²⁴. The term p_i^(Lreg) means the regression loss is activated only for positive anchors (p_i^ = 1) and is disabled otherwise (p_i^* = 0). The outputs of the cls and reg layers consist of {pi} and {ti} respectively.

Here, i is the index of an anchor in a mini-batch, and p_i is the predicted probability of anchor i being an object. The ground-truth label p_i ^ * is 1 if the anchor is positive, and 0 if the anchor is negative. t_i is a vector representing the 4 parameterized coordinates of the predicted bounding box, and t_i ^ * is that of the ground-truth box associated with a positive anchor. The classification loss Lcls is log loss over two classes (object vs. not object). For the regression loss, we use Lreg (t_i, t_i ^ *) = R (t_i - t_i ^ *), where R is the robust loss function (smooth L1) defined in ²²⁵. The term p_i ^ * (Lreg) means the regression loss is activated only for positive anchors (p_i ^ * = 1) and is disabled otherwise (p_i ^ * = 0). The outputs of the cls and reg layers consist of {pi} and {ti} respectively.

The two terms are normalized by Ncls and Nreg and weighted by a balancing parameter λ. In our current implementation (as in the released code), the cls term in Eqn.(1) is normalized by the mini-batch size (i.e., Ncls = 256) and the reg term is normalized by the number of anchor locations (i.e., Nreg ∼ 2, 400). By default we set λ = 10, and thus both cls and reg terms are roughly equally weighted. We show by experiments that the results are insensitive to the values of λ in a wide range (Table 9). We also note that the normalization as above is not required and could be simplified.

These terms are normalized by Ncls and Nreg and weighted by a balancing parameter λ. In our current implementation (as in the released code), the cls term in Eqn. (1) is normalized by the mini-batch size (i.e., Ncls = 256), and the reg term is normalized by the number of anchor locations (i.e., Nreg ~ 2, 400). By default we set λ = 10, and thus both cls and reg terms are roughly equally weighted. We show by experiments that the results are insensitive to the values of λ in a wide range (Table 9). Also, the normalization as above is not required and can be simplified.

For bounding box regression, we adopt the parameterizations of the 4 coordinates following ²²⁶:

For bounding box regression, we adopt the parameterizations of the 4 coordinates as follows ²²⁷:

where x, y, w, and h denote the box’s center coordinates and its width and height. Variables x, xa, and x^* are for the predicted box, anchor box, and groundtruth box respectively (likewise for y, w, h). This can be thought of as bounding-box regression from an anchor box to a nearby ground-truth box.

where x, y, w, and h denote the box's center coordinates and its width and height. Variables x, xa, and x ^ * are for the predicted box, anchor box, and groundtruth box respectively (likewise for y, w, h). This can be thought of as bounding-box regression from an anchor box to a nearby ground-truth box.

Nevertheless, our method achieves bounding-box regression by a different manner from previous RoIbased (Region of Interest) methods ²²⁸, ²²⁹. In ²³⁰, ²³¹, bounding-box regression is performed on features pooled from arbitrarily sized RoIs, and the regression weights are shared by all region sizes. In our formulation, the features used for regression are of the same spatial size (3 × 3) on the feature maps. To account for varying sizes, a set of k bounding-box regressors are learned. Each regressor is responsible for one scale and one aspect ratio, and the k regressors do not share weights. As such, it is still possible to predict boxes of various sizes even though the features are of a fixed size/scale, thanks to the design of anchors.

Nevertheless, our method achieves bounding-box regression in a different manner from previous RoIbased (Region of Interest) methods ²³², ²³³. In ²³⁴, ²³⁵, bounding-box regression is performed on features pooled from arbitrarily sized RoIs, and the regression weights are shared by all region sizes. In our formulation, the features used for regression are of the same spatial size (3 × 3) on the feature maps. To account for varying sizes, a set of k bounding-box regressors is learned. Each regressor is responsible for one scale and one aspect ratio, and the k regressors do not share weights. As such, it is still possible to predict boxes of various sizes even though the features are of a fixed size/scale, thanks to the design of anchors.

The RPN can be trained end-to-end by backpropagation and stochastic gradient descent (SGD) ²³⁶. We follow the “image-centric” sampling strategy from ²³⁷ to train this network. Each mini-batch arises from a single image that contains many positive and negative example anchors. It is possible to optimize for the loss functions of all anchors, but this will bias towards negative samples as they are dominate. Instead, we randomly sample 256 anchors in an image to compute the loss function of a mini-batch, where the sampled positive and negative anchors have a ratio of up to 1:1. If there are fewer than 128 positive samples in an image, we pad the mini-batch with negative ones.

The RPN can be trained end-to-end by backpropagation and stochastic gradient descent (SGD) ²³⁸. We follow the "image-centric" sampling strategy from ²³⁹ to train this network. Each mini-batch arises from a single image that contains many positive and negative example anchors. It is possible to optimize the loss functions of all anchors, but this will bias towards negative samples as they dominate. Instead, we randomly sample 256 anchors in an image to compute the loss function of a mini-batch, where the sampled positive and negative anchors have a ratio of 1:1. If there are fewer than 128 positive samples in an image, we pad the mini-batch with negative ones.

We randomly initialize all new layers by drawing weights from a zero-mean Gaussian distribution with standard deviation 0.01. All other layers (i.e., the shared convolutional layers) are initialized by pretraining a model for ImageNet classification ²⁴⁰, as is standard practice ²⁴¹. We tune all layers of the ZF net, and conv3 1 and up for the VGG net to conserve memory ²⁴². We use a learning rate of 0.001 for 60k mini-batches, and 0.0001 for the next 20k mini-batches on the PASCAL VOC dataset. We use a momentum of 0.9 and a weight decay of 0.0005 ²⁴³. Our implementation uses Caffe ²⁴⁴.

We randomly initialize all new layers by drawing weights from a zero-mean Gaussian distribution with standard deviation 0.01. All other layers (i.e., the shared convolutional layers) are initialized by pretraining a model for ImageNet classification ²⁴⁵, as is standard practice ²⁴⁶. We tune all layers of the ZF net, and conv3 1 and up for the VGG net to conserve memory ²⁴⁷. We use a learning rate of 0.001 for 60k mini-batches and 0.0001 for the next 20k mini-batches on the PASCAL VOC dataset. We use a momentum of 0.9 and a weight decay of 0.0005 ²⁴⁸. Our implementation uses Caffe ²⁴⁹.

Thus far we have described how to train a network for region proposal generation, without considering the region-based object detection CNN that will utilize these proposals. For the detection network, we adopt Fast R-CNN ²⁵⁰. Next we describe algorithms that learn a unified network composed of RPN and Fast R-CNN with shared convolutional layers (Figure 2).

So far we have described how to train a network for region proposal generation, without considering the region-based object detection CNN that will utilize these proposals. For the detection network, we adopt Fast R-CNN ²⁵¹. Next we describe algorithms that learn a unified network composed of RPN and Fast R-CNN with shared convolutional layers (Figure 2).

Both RPN and Fast R-CNN, trained independently, will modify their convolutional layers in different ways. We therefore need to develop a technique that allows for sharing convolutional layers between the two networks, rather than learning two separate networks. We discuss three ways for training networks with features shared:

Both RPN and Fast R-CNN, trained independently, will modify their convolutional layers in different ways. We therefore need to develop a technique that can learn two networks rather than sharing convolutional layers between the two networks. We discuss three ways of training networks with shared features:

(i) Alternating training. In this solution, we first train RPN, and use the proposals to train Fast R-CNN. The network tuned by Fast R-CNN is then used to initialize RPN, and this process is iterated. This is the solution that is used in all experiments in this paper.

(i) Alternating training. In this solution, we first train RPN, and use the proposals to train Fast R-CNN. The network tuned by Fast R-CNN is then used to initialize RPN, and this process is iterated. This is the solution used in all experiments in this paper.

(ii) Approximate joint training. In this solution, the RPN and Fast R-CNN networks are merged into one network during training as in Figure 2. In each SGD iteration, the forward pass generates region proposals which are treated just like fixed, pre-computed proposals when training a Fast R-CNN detector. The backward propagation takes place as usual, where for the shared layers the backward propagated signals from both the RPN loss and the Fast R-CNN loss are combined. This solution is easy to implement. But this solution ignores the derivative w.r.t. the proposal boxes’ coordinates that are also network responses, so is approximate. In our experiments, we have empirically found this solver produces close results, yet reduces the training time by about 25-50% comparing with alternating training. This solver is included in our released Python code.

(ii) Approximate joint training. In this solution, the RPN and Fast R-CNN networks are merged into one network during training as in Figure 2; in each SGD iteration, the forward pass generates region proposals, which are treated just like fixed, pre-computed proposals when training a Fast R-CNN detector. Backpropagation takes place as usual, where for the shared layers the backward-propagated signals from both the RPN loss and the Fast R-CNN loss are combined. This solution is easy to implement. But this solution ignores the derivative w.r.t. the proposal boxes' coordinates, which are also network responses, so it is approximate. In our experiments, we have empirically found that this solver produces close results, yet reduces the training time by about 25-50% compared with alternating training. This solver is included in our released Python code.

(iii) Non-approximate joint training. As discussed above, the bounding boxes predicted by RPN are also functions of the input. The RoI pooling layer ²⁵² in Fast R-CNN accepts the convolutional features and also the predicted bounding boxes as input, so a theoretically valid backpropagation solver should also involve gradients w.r.t. the box coordinates. These gradients are ignored in the above approximate joint training. In a non-approximate joint training solution, we need an RoI pooling layer that is differentiable w.r.t. the box coordinates. This is a nontrivial problem and a solution can be given by an “RoI warping” layer as developed in ²⁵³, which is beyond the scope of this paper.

(iii) Non-approximate joint training. As discussed above, the bounding boxes predicted by RPN are functions of the input. The RoI pooling layer ²⁵⁴ in Fast R-CNN accepts the convolutional features and also the predicted bounding boxes as input, so a theoretically valid backpropagation solver should involve gradients w.r.t. the box coordinates. These gradients are ignored in the above approximate joint training. In a non-approximate joint training solution, we need an RoI pooling layer that is differentiable w.r.t. the box coordinates. This is a trivial problem, and a solution can be given by an "RoI warping" layer as developed in ²⁵⁵, which is beyond the scope of this paper.

4-Step Alternating Training. In this paper, we adopt a pragmatic 4-step training algorithm to learn shared features via alternating optimization. In the first step, we train the RPN as described in Section 3.1.3. This network is initialized with an ImageNet-pre-trained model and fine-tuned end-to-end for the region proposal task. In the second step, we train a separate detection network by Fast R-CNN using the proposals generated by the step-1 RPN. This detection network is also initialized by the ImageNet-pre-trained model. At this point the two networks do not share convolutional layers. In the third step, we use the detector network to initialize RPN training, but we fix the shared convolutional layers and only fine-tune the layers unique to RPN. Now the two networks share convolutional layers. Finally, keeping the shared convolutional layers fixed, we fine-tune the unique layers of Fast R-CNN. As such, both networks share the same convolutional layers and form a unified network. A similar alternating training can be run for more iterations, but we have observed negligible improvements.

4-Step Alternating Training. In this paper, we adopt a practical 4-step training algorithm to learn shared features via alternating optimization. In the first step, we train the RPN as described in Section 3.1.3. This network is initialized with an ImageNet-pre-trained model and fine-tuned end-to-end for the region proposal task. In the second step, we train a separate detection network by Fast R-CNN using the proposals generated by the step-1 RPN. This detection network is also initialized by the ImageNet-pre-trained model. At this point the two networks do not share convolutional layers. In the third step, we use the detector network to initialize RPN training, but we fix the shared convolutional layers and fine-tune only the layers unique to RPN. Now the two networks share convolutional layers. Finally, keeping the shared convolutional layers fixed, we fine-tune the unique layers of Fast R-CNN. As such, both networks share the same convolutional layers and form a unified network. A similar alternating training can be run for more iterations, but we have observed negligible improvements.

We train and test both region proposal and object detection networks on images of a single scale ²⁵⁶, ²⁵⁷. We re-scale the images such that their shorter side is s = 600 pixels ²⁵⁸. Multi-scale feature extraction (using an image pyramid) may improve accuracy but does not exhibit a good speed-accuracy trade-off ²⁵⁹. On the re-scaled images, the total stride for both ZF and VGG nets on the last convolutional layer is 16 pixels, and thus is ∼10 pixels on a typical PASCAL image before resizing (∼500×375). Even such a large stride provides good results, though accuracy may be further improved with a smaller stride.

We train and test both region proposal and object detection networks on images of a single scale ²⁶⁰, ²⁶¹. We re-scale the images such that their shorter side is s = 600 pixels ²⁶². Multi-scale feature extraction (using an image pyramid) may improve accuracy but does not yield a good speed-accuracy trade-off ²⁶³. On the re-scaled images, the total stride for both ZF and VGG nets on the last convolutional layer is 16 pixels, and thus is ~10 pixels on a typical PASCAL image before resizing (~500 × 375). Even such a large stride provides good results, though accuracy may be improved with a smaller stride.

For anchors, we use 3 scales with box areas of 128^2, 256^2 , and 512^2 pixels, and 3 aspect ratios of 1:1, 1:2, and 2:1. These hyper-parameters are not carefully chosen for a particular dataset, and we provide ablation experiments on their effects in the next section. As discussed, our solution does not need an image pyramid or filter pyramid to predict regions of multiple scales, saving considerable running time. Figure 3 (right) shows the capability of our method for a wide range of scales and aspect ratios. Table 1 shows the learned average proposal size for each anchor using the ZF net. We note that our algorithm allows predictions that are larger than the underlying receptive field. Such predictions are not impossible—one may still roughly infer the extent of an object if only the middle of the object is visible.

For anchors, we use 3 scales with box areas of 128^2, 256^2, and 512^2 pixels, and 3 aspect ratios of 1:1, 1:2, and 2:1. These hyper-parameters are not carefully chosen for a particular dataset, and we provide ablation experiments on their effects in the next section. As discussed, our solution does not need an image pyramid or filter pyramid to predict regions of multiple scales, saving considerable running time. Figure 3 (right) shows the capability of our method for a wide range of scales and aspect ratios. Table 1 shows the learned average proposal size for each anchor using the ZF net. Our algorithm allows predictions that are larger than the underlying receptive field. Such predictions are not impossible—one may still roughly infer the extent of an object if only the middle of the object is visible.

The anchor boxes that cross image boundaries need to be handled with care. During training, we ignore all cross-boundary anchors so they do not contribute to the loss. For a typical 1000 × 600 image, there will be roughly 20000 (≈ 60 × 40 × 9) anchors in total. With the cross-boundary anchors ignored, there are about 6000 anchors per image for training. If the boundary-crossing outliers are not ignored in training, they introduce large, difficult to correct error terms in the objective, and training does not converge. During testing, however, we still apply the fully convolutional RPN to the entire image. This may generate crossboundary proposal boxes, which we clip to the image boundary.

The anchor boxes that cross image boundaries need to be handled with care. During training, we ignore cross-boundary anchors so they do not contribute to the loss. For a typical 1000 × 600 image, there will be roughly 20000 (≈ 60 × 40 × 9) anchors in total. With the cross-boundary anchors ignored, there are about 6000 anchors per image for training. If the boundary-crossing outliers are not ignored in training, they introduce large, difficult-to-correct error terms in the objective, and training does not converge. During testing, however, we still apply the fully convolutional RPN to the entire image. This may generate crossboundary proposal boxes, which we clip to the image boundary.

Some RPN proposals highly overlap with each other. To reduce redundancy, we adopt non-maximum suppression (NMS) on the proposal regions based on their cls scores. We fix the IoU threshold for NMS at 0.7, which leaves us about 2000 proposal regions per image. As we will show, NMS does not harm the ultimate detection accuracy, but substantially reduces the number of proposals. After NMS, we use the top-N ranked proposal regions for detection. In the following, we train Fast R-CNN using 2000 RPN proposals, but evaluate different numbers of proposals at test-time.

Some RPN proposals highly overlap with each other. To reduce redundancy, we adopt non-maximum suppression (NMS) on the proposal regions based on their cls scores. We fix the IoU threshold for NMS at 0.7, which leaves us about 2,000 proposal regions per image. As we will show, NMS does not harm the ultimate detection accuracy, but substantially reduces the number of proposals. After NMS, we use the top-N ranked proposal regions for detection. In the following, we train Fast R-CNN using 2000 RPN proposals, but evaluate different numbers of proposals at test time.

We have presented RPNs for efficient and accurate region proposal generation. By sharing convolutional features with the down-stream detection network, the region proposal step is nearly cost-free. Our method enables a unified, deep-learning-based object detection system to run at near real-time frame rates. The learned RPN also improves region proposal quality and thus the overall object detection accuracy.

We have presented LPNs for efficient and accurate region proposal generation. By sharing convolutional features with the downstream detection network, the region proposal step is nearly cost-free. Our method enables a unified, deep-learning-based object detection system to run at near real-time frame rates. The learned RPN improves region proposal quality and thus the overall object detection accuracy.