R-CNN:Paper

Object detection performance, as measured on the canonical PASCAL VOC dataset, has plateaued in the last few years. The best-performing methods are complex ensemble systems that typically combine multiple low-level image features with high-level context. In this paper, we propose a simple and scalable detection algorithm that improves mean average precision (mAP) by more than 30% relative to the previous best result on VOC 2012—achieving a mAP of 53.3%. Our approach combines two key insights: (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals in order to localize and segment objects and (2) when labeled training data is scarce, supervised pre-training for an auxiliary task, followed by domain-specific fine-tuning, yields a significant performance boost. Since we combine region proposals with CNNs, we call our method R-CNN: Regions with CNN features. We also compare R-CNN to OverFeat, a recently proposed sliding-window detector based on a similar CNN architecture. We find that R-CNN outperforms OverFeat by a large margin on the 200-class ILSVRC2013 detection dataset. Source code for the complete system is available at http://www.cs.berkeley.edu/˜rbg/rcnn

Object detection performance, measured on the standard PASCAL VOC dataset, has plateaued in recent years. The best-performing methods are complex ensemble systems that typically combine multiple low-level image features with high-level context. In this paper, we propose a simple and scalable detection algorithm that improves mean average precision (MAP) by more than 30% over the previous best result, achieving a map of 53.3% on volatile organic compounds (VOC) in 2012. To localize, segment objects and (2) when labeled training data is scarce, supervised pre-training (1) one can apply high-capacity convolutional neural networks (CNNs) to bottom-up region proposals: our approach combines two important insights, for an auxiliary task followed by domain-specific fine-tuning, a significant performance boost can be obtained. Regions with CNN features: because we combine region proposals with CNNs, we call our method R-local. We also compare with R-CNN OverFeat, a sliding-window detector recently proposed based on a similar CNN architecture. We find that R-CNN outperforms OverFeat by a large margin on the 200-grade ILSVRC2013 detection dataset. The source code for the complete system can be found at http://www.cs.berkeley.edu/~rbg/rcnn

Features matter. The last decade of progress on various visual recognition tasks has been based considerably on the use of SIFT [29] and HOG [7]. But if we look at performance on the canonical visual recognition task, PASCAL VOC object detection [15], it is generally acknowledged that progress has been slow during 2010-2012, with small gains obtained by building ensemble systems and employing minor variants of successful methods.

Features are important. The research of the past 10 years on various visual recognition tasks has relied considerably on the use of SIFT [29] and HOG [7]. However, looking at performance on the standard visual recognition task, PASCAL VOC object detection [15], it is generally acknowledged that progress was slow during 2010-2012, with small gains obtained by building ensemble systems and using small variants of successful methods.

SIFT and HOG are blockwise orientation histograms, a representation we could associate roughly with complex cells in V1, the first cortical area in the primate visual pathway. But we also know that recognition occurs several stages downstream, which suggests that there might be hierarchical, multi-stage processes for computing features that are even more informative for visual recognition.

SIFT and HOG are blockwise orientation histograms that can be roughly associated with complex cells in V1, the first cortical area of the primate visual pathway. However, we know that recognition occurs several stages downstream. This means that there are hierarchical, multi-stage processes for computing features that are more informative for visual recognition.

Fukushima’s “neocognitron” [19], a biologicallyinspired hierarchical and shift-invariant model for pattern recognition, was an early attempt at just such a process. The neocognitron, however, lacked a supervised training algorithm. Building on Rumelhart et al. [33], LeCun et al. [26] showed that stochastic gradient descent via backpropagation was effective for training convolutional neural networks (CNNs), a class of models that extend the neocognitron.

Fukushima's "neocognitron" [19] is a biologically stimulated hierarchical and shift-invariant model for pattern recognition, an early attempt at such a process. However, the neocognitron lacked a supervised training algorithm. Rumelhart et al. [33], LeCun et al. [26] showed that stochastic gradient descent via backpropagation is effective for training CNNs (convolutional neural networks), models that extend the neocognitron.

CNNs saw heavy use in the 1990s (e.g., [27]), but then fell out of fashion with the rise of support vector machines. In 2012, Krizhevsky et al. [25] rekindled interest in CNNs by showing substantially higher image classification accuracy on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [9, 10]. Their success resulted from training a large CNN on 1.2 million labeled images, together with a few twists on LeCun’s CNN (e.g., max(x, 0) rectifying non-linearities and “dropout” regularization).

CNNs showed excessive use in the 1990s (e.g., [27]), but went out of fashion with the rise of support vector machines. In 2012, Krizhevsky et al. [25] showed renewed interest in CNNs by demonstrating considerably higher image classification accuracy on the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [9, 10]. Their success is the result of training a large CNN together with 1.2 million labeled images, along with a few twists on LeCun's CNN (e.g., max (x, 0) rectifying non-linearity and "dropout" regularization).

The significance of the ImageNet result was vigorously debated during the ILSVRC 2012 workshop. The central issue can be distilled to the following: To what extent do the CNN classification results on ImageNet generalize to object detection results on the PASCAL VOC Challenge?

The importance of the ImageNet results was actively discussed at the ILSVRC 2012 workshop. The central issue can be extracted as follows: to what extent can CNN classification results on ImageNet generalize to object detection results on the PASCAL VOC Challenge?

We answer this question by bridging the gap between image classification and object detection. This paper is the first to show that a CNN can lead to dramatically higher object detection performance on PASCAL VOC as compared to systems based on simpler HOG-like features. To achieve this result, we focused on two problems: localizing objects with a deep network and training a high-capacity model with only a small quantity of annotated detection data.

We answer this question by bridging the gap between image classification and object detection. This paper is the first to show that a CNN can lead to much higher object detection performance on PASCAL VOC compared to systems based on simple HOG-like features. To achieve this result, we focused on two problems. We localized objects with a deep network and trained a high-capacity model with a small amount of annotated detection data.

Unlike image classification, detection requires localizing (likely many) objects within an image. One approach frames localization as a regression problem. However, work from Szegedy et al. [38], concurrent with our own, indicates that this strategy may not fare well in practice (they report a mAP of 30.5% on VOC 2007 compared to the 58.5% achieved by our method). An alternative is to build a sliding-window detector. CNNs have been used in this way for at least two decades, typically on constrained object categories, such as faces [32, 40] and pedestrians [35]. In order to maintain high spatial resolution, these CNNs typically only have two convolutional and pooling layers. We also considered adopting a sliding-window approach. However, units high up in our network, which has five convolutional layers, have very large receptive fields (195 × 195 pixels) and strides (32×32 pixels) in the input image, which makes precise localization within the sliding-window paradigm an open technical challenge

Unlike image classification, objects (in many cases) must be localized within an image. One approach frames localization as a regression problem. However, Szegedy et al. [38], concurrently with ourselves, indicates that this strategy may not actually run. (Reports a mAP of 30.5% on VOC 2007 compared to the 58.5% achieved by our method). Another method is to build a sliding-window detector. CNNs have commonly been used for at least 20 years on constrained object categories such as faces [32,40]. And pedestrians [35]. To maintain high spatial resolution, these CNNs typically have only two convolution and pooling layers. We also considered adopting a sliding-window approach. However, the high units of our network, which has five convolutional layers, have very large receptive fields (195 × 195 pixels) and strides (32 × 32 pixels) in the input image, so precise localization within the sliding-window paradigm is possible. An open technical challenge

Instead, we solve the CNN localization problem by operating within the “recognition using regions” paradigm [21], which has been successful for both object detection [39] and semantic segmentation [5]. At test time, our method generates around 2000 category-independent region proposals for the input image, extracts a fixed-length feature vector from each proposal using a CNN, and then classifies each region with category-specific linear SVMs. We use a simple technique (affine image warping) to compute a fixed-size CNN input from each region proposal, regardless of the region’s shape. Figure 1 presents an overview of our method and highlights some of our results. Since our system combines region proposals with CNNs, we dub the method R-CNN: Regions with CNN features.

Instead, we solve the CNN localization problem by operating within the "regions using recognition" paradigm [21], which has been successful for both object detection [39] and semantic segmentation [5]. At test time, our method generates about 2000 category-independent region proposals for the input image, extracts a fixed-length feature vector from each proposal using a CNN, and then classifies each region with category-specific linear SVMs. We use a simple technique (appeal image warping) to compute a fixed-size CNN input from each region proposal regardless of the region's shape. Figure 1 presents an overview of our method and highlights some results. Because the Google system combines region proposals with CNNs, we gave it the title R-CNN: the region method with CNN features.

In this updated version of this paper, we provide a headto-head comparison of R-CNN and the recently proposed OverFeat [34] detection system by running R-CNN on the 200-class ILSVRC2013 detection dataset. OverFeat uses a sliding-window CNN for detection and until now was the best performing method on ILSVRC2013 detection. We show that R-CNN significantly outperforms OverFeat, with a mAP of 31.4% versus 24.3%.

In the updated version of this white paper, we provide a head - head comparison of R-CNN and the recently proposed OverFeat [34] detection system by running R-CNN on the 200-class ILSVRC2013 detection dataset. OverFeat uses a sliding-window CNN for detection and until now was the best-performing method on ILSVRC2013 detection. We show that R-CNN is far superior to OverFeat, with a mAP of 31.4% compared to 24.3%.

A second challenge faced in detection is that labeled data is scarce and the amount currently available is insufficient for training a large CNN. The conventional solution to this problem is to use unsupervised pre-training, followed by supervised fine-tuning (e.g., [35]). The second principle contribution of this paper is to show thatsupervised pre-training on a large auxiliary dataset (ILSVRC), followed by domainspecific fine-tuning on a small dataset (PASCAL), is an effective paradigm for learning high-capacity CNNs when data is scarce. In our experiments, fine-tuning for detection improves mAP performance by 8 percentage points. After fine-tuning, our system achieves a mAP of 54% on VOC 2010 compared to 33% for the highly-tuned, HOG-based deformable part model (DPM) [17, 20]. We also point readers to contemporaneous work by Donahue et al. [12], who show that Krizhevsky’s CNN can be used (without finetuning) as a blackbox feature extractor, yielding excellent performance on several recognition tasks including scene classification, fine-grained sub-categorization, and domain adaptation.

The second challenge faced in detection is that labeled data is scarce and the sum currently available is not enough to train a large CNN. The conventional solution to this problem is to use unsupervised pre-training followed by supervised fine-tuning (e.g., [35]). The second basic contribution of this paper is to show supervised pre-training on a large auxiliary dataset (ILSVRC) and domain-specific fine-tuning on a small dataset (PASCAL), and it is an effective paradigm for learning high-capacity CNNs when data is. Scarce. In our experiments, fine-tuning for detection improves mAP performance by 8%. After fine-tuning, our system achieves a mAP of 54% on VOC 2010, compared to 33% for the highly tuned HOG-based deformable parts model (DPM) [17, 20]. We also point readers to the concurrent work of Donahue et al. [12] shows that Krizhevsky's CNN can be used as a black-box feature extractor, providing excellent performance on several recognition tasks including scene classification, fine-grained sub-categorization, and domain adaptation.

Our system is also quite efficient. The only class-specific computations are a reasonably small matrix-vector product and greedy non-maximum suppression. This computational property follows from features that are shared across all categories and that are also two orders of magnitude lowerdimensional than previously used region features (cf. [39]).

Our system is also very efficient. The class-specific computations are a reasonably small matrix - vector product and non-greedy non - maximum suppression. This computational property derives from features that are shared across all categories and have a dimension 2 times smaller than previously used region features (cf. [39]).

Understanding the failure modes of our approach is also critical for improving it, and so we report results from the detection analysis tool of Hoiem et al. [23]. As an immediate consequence of this analysis, we demonstrate that a simple bounding-box regression method significantly reduces mislocalizations, which are the dominant error mode.

Understanding the failure modes of our approach is also important for improving it. So we report the results of the detection analysis tool of Hoiem et al. [23]. As an immediate result of this analysis, we show that a simple bounding-box regression method greatly reduces mislocalizations, the dominant error mode.

Before developing technical details, we note that because R-CNN operates on regions it is natural to extend it to the task of semantic segmentation. With minor modifications, we also achieve competitive results on the PASCAL VOC segmentation task, with an average segmentation accuracy of 47.9% on the VOC 2011 test set.

Before developing the technical details, because R-CNN operates on regions, it is natural to extend it to the semantic segmentation task. Through minor modifications, we also obtain competitive results on the PASCAL VOC segmentation task, with an average segmentation accuracy of 47.9% on the VOC 2011 test set.

Figure 1: Object detection system overview. Our system (1) takes an input image, (2) extracts around 2000 bottom-up region proposals, (3) computes features for each proposal using a large convolutional neural network (CNN), and then (4) classifies each region using class-specific linear SVMs. R-CNN achieves a mean average precision (mAP) of 53.7% on PASCAL VOC 2010. For comparison, [39] reports 35.1% mAP using the same region proposals, but with a spatial pyramid and bag-of-visual-words approach. The popular deformable part models perform at 33.4%. On the 200-class ILSVRC2013 detection dataset, R-CNN’s mAP is 31.4%, a large improvement over OverFeat [34], which had the previous best result at 24.3%.

Our object detection system consists of three modules. The first generates category-independent region proposals. These proposals define the set of candidate detections available to our detector. The second module is a large convolutional neural network that extracts a fixed-length feature vector from each region. The third module is a set of classspecific linear SVMs. In this section, we present our design decisions for each module, describe their test-time usage, detail how their parameters are learned, and show detection results on PASCAL VOC 2010-12 and on ILSVRC2013.

Our object detection system consists of three modules. The first generates category-independent region proposals. These proposals define the set of candidate detections usable by our detector. The second module is a large-scale convolutional neural network that extracts a fixed-length feature vector from each region. The third module is a set of class-specific linear SVMs. In this section, we present our design decisions for each module, explain how their own parameters were learned, their own test-time usage, and details, and show detection results on PASCAL VOC 2010-12 and ILSVRC 2013.

Region proposals. A variety of recent papers offer methods for generating category-independent region proposals. Examples include: objectness [1], selective search [39], category-independent object proposals [14], constrained parametric min-cuts (CPMC) [5], multi-scale combinatorial grouping [3], and Cires¸an et al. [6], who detect mitotic cells by applying a CNN to regularly-spaced square crops, which are a special case of region proposals. While R-CNN is agnostic to the particular region proposal method, we use selective search to enable a controlled comparison with prior detection work (e.g., [39, 41]).

Region proposals. A variety of recent papers provide methods for generating category-independent region proposals. Examples include: objectness [1], selective search [39], category-independent object proposals [14], constrained parametric minute cuts (CPMC) [5], multi-scale combinatorial grouping [3], and Cires¸an et al. [6], people who detect mitotic cells by applying a CNN regularly space square crops (as in farm crops), which are a special case of region proposals. Although R-CNN is agnostic to the particular region proposal method, we use selective search to enable a controlled comparison with previous detection work (e.g., [39, 41]).

Feature extraction. We extract a 4096-dimensional feature vector from each region proposal using the Caffe [24] implementation of the CNN described by Krizhevsky et al. [25]. Features are computed by forward propagating a mean-subtracted 227 × 227 RGB image through five convolutional layers and two fully connected layers. We refer readers to [24, 25] for more network architecture details.

Feature extraction. We decompress a 4096-dimensional feature vector from each region proposal using the [24] implementation of the CNN described by CAFFE Krizhevsky et al. [25]. Features are computed by propagating a mean-subtracted 227 × 227 RGB image forward through five convolutional layers and two fully connected layers. Please refer readers to [24, 25] for more network architecture details.

In order to compute features for a region proposal, we must first convert the image data in that region into a form that is compatible with the CNN (its architecture requires inputs of a fixed 227 × 227 pixel size). Of the many possible transformations of our arbitrary-shaped regions, we opt for the simplest. Regardless of the size or aspect ratio of the candidate region, we warp all pixels in a tight bounding box around it to the required size. Prior to warping, we dilate the tight bounding box so that at the warped size there are exactly p pixels of warped image context around the original box (we use p = 16). Figure 2 shows a random sampling of warped training regions. Alternatives to warping are discussed in Appendix A.

To compute region proposal features, we first convert the image data of that region into a form compatible with the CNN (its structure requires input of a fixed 227 × 227 pixel size). Among the many variations of our arbitrary-shaped regions, we choose the simple one. Regardless of the size or aspect ratio of the candidate region, we warp all pixels within an approximately tight bounding box to the required size. Before warping, so that there are page pixels of warped image context exactly around the original box at the warped size, we dilate the tight bounding box (we use P = 16). Figure 2 shows a random sampling of warped training regions. Alternatives to warping are described in Appendix A

At test time, we run selective search on the test image to extract around 2000 region proposals (we use selective search’s “fast mode” in all experiments). We warp each proposal and forward propagate it through the CNN in order to compute features. Then, for each class, we score each extracted feature vector using the SVM trained for that class. Given all scored regions in an image, we apply a greedy non-maximum suppression (for each class independently) that rejects a region if it has an intersection-overunion (IoU) overlap with a higher scoring selected region larger than a learned threshold.

At test time, we run selective search on the test image to extract about 2000 region proposals (we use selective search's "fast mode" in all experiments). We warp each proposal and propagate it forward through the CNN to compute features. Then, for each class, we score each extracted feature vector using the SVM trained for that class. Given all scored regions of the image, we apply non-greedy maximum suppression (independently for each class) that rejects a region if it has a cross-over labor union (you) overlay with a higher-scoring selected region larger than the learned threshold.

Run-time analysis. Two properties make detection efficient. First, all CNN parameters are shared across all categories. Second, the feature vectors computed by the CNN are low-dimensional when compared to other common approaches, such as spatial pyramids with bag-of-visual-word encodings. The features used in the UVA detection system [39], for example, are two orders of magnitude larger than ours (360k vs. 4k-dimensional).

Run-time analysis. Two properties make detection efficient. First, all CNN parameters are shared across all categories. Second, the feature vectors computed by the CNN are low-dimensional compared to other common approaches, such as spatial pyramids with pouch-of-visual-word encodings. The features used in the UVA detection system [39], for example, are larger than ours by a 2nd-order magnitude (360K vs. 4K dimensions).

The result of such sharing is that the time spent computing region proposals and features (13s/image on a GPU or 53s/image on a CPU) is amortized over all classes. The only class-specific computations are dot products between features and SVM weights and non-maximum suppression. In practice, all dot products for an image are batched into a single matrix-matrix product. The feature matrix is typically 2000×4096 and the SVM weight matrix is 4096×N, where N is the number of classes.

The result of such sharing is that the time spent computing region proposals and features (13S/image on GPU or 53S/image on CPU) is amortized over all classes. The only class-specific computations are dot products(goods) between features and SVM weights, and non-maximum suppression. In practice, all dot products for an image are batched into a single matrix-matrix product(goods). The feature matrix is 2000 × 4096, and the SVM weight matrix is typically 4096 × N, where N is the number of classes.

This analysis shows that R-CNN can scale to thousands of object classes without resorting to approximate techniques, such as hashing. Even if there were 100k classes, the resulting matrix multiplication takes only 10 seconds on a modern multi-core CPU. This efficiency is not merely the result of using region proposals and shared features. The UVA system, due to its high-dimensional features, would be two orders of magnitude slower while requiring 134GB of memory just to store 100k linear predictors, compared to just 1.5GB for our lower-dimensional features.

This analysis shows that R-CNN can scale to thousands of object classes without relying on approximate techniques such as hashing. Even if there were 100K classes, the resulting matrix multiplication takes 10 seconds on a modern multi-core CPU. This efficiency is not merely the result of using region proposals and shared features. Compared to just 1.5GB for our low-dimensional features, while requiring memory to store just 100K linear predictors, the 134-gigabyte UVA system would be slower, by a 2nd order of magnitude, due to its high-dimensional features.

It is also interesting to contrast R-CNN with the recent work from Dean et al. on scalable detection using DPMs and hashing [8]. They report a mAP of around 16% on VOC 2007 at a run-time of 5 minutes per image when introducing 10k distractor classes. With our approach, 10k detectors can run in about a minute on a CPU, and because no approximations are made mAP would remain at 59% (Section 3.2).

It is also interesting to contrast R-CNN with the recent work of Dean et al. et al. Hashing [8] using DPMS for scalable detection. When introducing 10,000 distractor classes, they report a map of about 16% on VOC 2007 at a run time of 5 minutes per image. With our approach, 10,000 detectors can run in about a minute on a CPU, and because no more approximations are made, the map would remain at 59% (Section 3.2).

Supervised pre-training. We discriminatively pre-trained the CNN on a large auxiliary dataset (ILSVRC2012 classification) using image-level annotations only (boundingbox labels are not available for this data). Pre-training was performed using the open source Caffe CNN library [24]. In brief, our CNN nearly matches the performance of Krizhevsky et al. [25], obtaining a top-1 error rate 2.2 percentage points higher on the ILSVRC2012 classification validation set. This discrepancy is due to simplifications in the training process.

Supervising pre-education. We pre-trained the CNN on a large auxiliary dataset (ILSVRC2012 classification) using only differentiated image-level annotations (BoundingBox labels cannot use this data). Pre-education was done using the open-source CAFFE CNN library [24]. In short, our CNN nearly matches the performance of Krizhevsky et al. [25], and a top-1 error rate 2.2% points higher can be obtained on the ILSVRC2012 classification validation set. This discrepancy is due to simplification of the education process.

Domain-specific fine-tuning. To adapt our CNN to the new task (detection) and the new domain (warped proposal windows), we continue stochastic gradient descent (SGD) training of the CNN parameters using only warped region proposals. Aside from replacing the CNN’s ImageNetspecific 1000-way classification layer with a randomly initialized (N + 1)-way classification layer (where N is the number of object classes, plus 1 for background), the CNN architecture is unchanged. For VOC, N = 20 and for ILSVRC2013, N = 200. We treat all region proposals with ≥ 0.5 IoU overlap with a ground-truth box as positives for that box’s class and the rest as negatives. We start SGD at a learning rate of 0.001 (1/10th of the initial pre-training rate), which allows fine-tuning to make progress while not clobbering the initialization. In each SGD iteration, we uniformly sample 32 positive windows (over all classes) and 96 background windows to construct a mini-batch of size 128. We bias the sampling towards positive windows because they are extremely rare compared to background.

Domain-specific fine-tuning. To adapt our CNN to the new task (detection) and the new domain (warped proposal windows), we continue stochastic gradient descent (SGD) training of the CNN parameters using warped region proposals. Aside from replacing the CNN's ImageNetspecific 1000-way classification layer with a randomly initialized (N + 1)-way classification layer (N is the number of object classes plus 1 for background), the CNN structure is unchanged. For VOC, = 20, N for ILSVRC 2013 N200. We treat all region proposals with ≥ 0.5 IOU overlap with a ground truth box as positives of the box's class and the rest as negatives. We start SGD at a learning rate of 0.001 (1/10 of the initial pre-training rate), which lets fine-tuning make progress without touching the initialization. Because they are very rare compared to the background, in each SGD iteration we bias the sampling toward positive windows, uniformly over 32 positive windows (over all classes) and 96 background windows, to sample in order to construct a mini-batch of size 128.

Object category classifiers. Consider training a binary classifier to detect cars. It’s clear that an image region tightly enclosing a car should be a positive example. Similarly, it’s clear that a background region, which has nothing to do with cars, should be a negative example. Less clear is how to label a region that partially overlaps a car. We resolve this issue with an IoU overlap threshold, below which regions are defined as negatives. The overlap threshold, 0.3, was selected by a grid search over {0, 0.1, . . . , 0.5} on a validation set. We found that selecting this threshold carefully is important. Setting it to 0.5, as in [39], decreased mAP by 5 points. Similarly, setting it to 0 decreased mAP by 4 points. Positive examples are defined simply to be the ground-truth bounding boxes for each class.

Object category classification. Try training a binary classifier that detects cars. It is clear that an image region tightly enclosing a car should be a positive example. Similarly, it is clear that a background region unrelated to cars should exclude the example. Less clear is how to label a region that partially overlaps a car. We solve this problem with an IOU overlap threshold, below which regions are defined as negatives. The overlap threshold of 0.3 was selected by a grid search over {0, 0.1. . . validation set, 0.5}. We found that selecting this threshold carefully is important. Setting it to 0.5 as in [39] reduced the map by 5 points. Similarly, setting it to 0 decreased by 4 points. Positive examples are simply defined as the ground truth of each class's bounding box.

Once features are extracted and training labels are applied, we optimize one linear SVM per class. Since the training data is too large to fit in memory, we adopt the standard hard negative mining method [17, 37]. Hard negative mining converges quickly and in practice mAP stops increasing after only a single pass over all images.

Once features are extracted and education labels are applied, we can optimize one linear SVM per class. Because the training data is too large compared to memory, we adopt the standard hard negative mining method [17, 37]. Hard negative mining quickly and in practice the convergence map does not increase a single pass over all images.

In Appendix B we discuss why the positive and negative examples are defined differently in fine-tuning versus SVM training. We also discuss the trade-offs involved in training detection SVMs rather than simply using the outputs from the final softmax layer of the fine-tuned CNN.

In Appendix B we discuss why anode and cathode examples are defined differently in fine-tuning compared to SVM training. We also explain the pros and cons involved in training detection SVMs rather than simply using the output from the final softmax layer of the fine-tuned CNN.

Following the PASCAL VOC best practices [15], we validated all design decisions and hyperparameters on the VOC 2007 dataset (Section 3.2). For final results on the VOC 2010-12 datasets, we fine-tuned the CNN on VOC 2012 train and optimized our detection SVMs on VOC 2012 trainval. We submitted test results to the evaluation server only once for each of the two major algorithm variants (with and without bounding-box regression).

Following PASCAL VOC best practices [15], we validated all design decisions and hyperparameters on the VOC 2007 dataset (Section 3.2). For the final results on the VOC 2010-12 datasets, we fine-tuned the CNN on the VOC 2012 train (railway) and optimized our detection SVMs on VOC 2012 trainval. We submitted the test results to the server evaluation only once for each of the two algorithm variants without (with bounding-box regression).

Table 1 shows complete results on VOC 2010. We compare our method against four strong baselines, including SegDPM [18], which combines DPM detectors with the output of a semantic segmentation system [4] and uses additional inter-detector context and image-classifier rescoring. The most germane comparison is to the UVA system from Uijlings et al. [39], since our systems use the same region proposal algorithm. To classify regions, their method builds a four-level spatial pyramid and populates it with densely sampled SIFT, Extended OpponentSIFT, and RGBSIFT descriptors, each vector quantized with 4000-word codebooks. Classification is performed with a histogram intersection kernel SVM. Compared to their multi-feature, non-linear kernel SVM approach, we achieve a large improvement in mAP, from 35.1% to 53.7% mAP, while also being much faster (Section 2.2). Our method achieves similar performance (53.3% mAP) on VOC 2011/12 test.

Table 1 shows full results on VOC 2010, comparing our method against 4 strong baselines, including SegDPM [18], which combines DPM detectors with the output of a semantic segmentation system [4] and uses additional inter-detector context and image classification, rescoring. The most closely related comparison is the UVA system from Uijlings et al. [39], because our system uses the same region proposal algorithm. To classify regions, their method builds a 4-level spatial pyramid and fills it with densely sampled SIFT, extended opponent's SIFT, and RGB SIFT technicians, each vector quantized into 4000-word codebooks. Classification is performed with a histogram intersection kernel SVM. Compared to their multi-feature nonlinear kernel SVM method, while being much faster (Section 2.2), we achieve a large improvement of the map from 35.1% to 53.7% map. Our method achieves similar performance (53.3% map) on the volatile organic compounds (VOC) 2011/12 test.

We ran R-CNN on the 200-class ILSVRC2013 detection dataset using the same system hyperparameters that we used for PASCAL VOC. We followed the same protocol of submitting test results to the ILSVRC2013 evaluation server only twice, once with and once without bounding-box regression.

We ran R-CNN on the 200-grade ILSVRC 2013 detection dataset using the same system hyperparameters we used for PASCAL VOC. We followed the same protocol of submitting test results to the ILSVRC 2013 evaluation server only twice, once at a time, without bounding box regression.

Figure 3 compares R-CNN to the entries in the ILSVRC 2013 competition and to the post-competition OverFeat result [34]. R-CNN achieves a mAP of 31.4%, which is significantly ahead of the second-best result of 24.3% from OverFeat. To give a sense of the AP distribution over classes, box plots are also presented and a table of perclass APs follows at the end of the paper in Table 8. Most of the competing submissions (OverFeat, NEC-MU, UvAEuvision, Toronto A, and UIUC-IFP) used convolutional neural networks, indicating that there is significant nuance in how CNNs can be applied to object detection, leading to greatly varying outcomes.

Figure 3 compares R-CNN against the entries in the ILSVRC 2013 competition and the post-competition OverFeat result [34]. R-CNN can obtain a map of 31.4%, well ahead of the second-best result of 24.3% from OverFeat. To give a sense of the AP distribution over classes, box plots are also provided, and a table of perclass access points follows at the end of the sheet of paper in Table, and most of the competing submissions (OverFeat, NEC-MU, UvAEuvision, Toronto 8., and UIUC-IFP) used convolutional neural networks, indicating that there is considerable nuance in the way CNNs can be applied to detection objects, leading to greatly varying results.

In Section 4, we give an overview of the ILSVRC2013 detection dataset and provide details about choices that we made when running R-CNN on it.

In Section 4, we provide an overview of the ILSVRC2013 detection dataset and give detailed information about the choices we made when running R-CNN on it.

First-layer filters can be visualized directly and are easy to understand [25]. They capture oriented edges and opponent colors. Understanding the subsequent layers is more challenging. Zeiler and Fergus present a visually attractive deconvolutional approach in [42]. We propose a simple (and complementary) non-parametric method that directly shows what the network learned.

First-layer filters can be directly visualized and made easy to understand [25]. They capture oriented edges and relative colors. Understanding the next layers is more difficult. ZEILER and Fergus present a visually attractive deconvolutional approach in [42]. We propose a simple (complementary) nonparametric method that directly shows what the network has learned.

The idea is to single out a particular unit (feature) in the network and use it as if it were an object detector in its own right. That is, we compute the unit’s activations on a large set of held-out region proposals (about 10 million), sort the proposals from highest to lowest activation, perform nonmaximum suppression, and then display the top-scoring regions. Our method lets the selected unit “speak for itself” by showing exactly which inputs it fires on. We avoid averaging in order to see different visual modes and gain insight into the invariances computed by the unit.

The idea is to single out a particular part (feature) in the network and use it as if it were itself an object detection part. That is, we compute the device's activations on a large set of open-out region proposals (about 10 million dollars), sort the proposals from highest to lowest activation, perform non-maximum suppression, and display the top-scoring regions. Our method lets the selected device "talk about itself" by accurately showing the inputs on fire. We avoid averaging in order to see different visual modes and gain insight into the invariances computed per unit.

We visualize units from layer pool5 , which is the maxpooled output of the network’s fifth and final convolutional layer. The pool5 feature map is 6 × 6 × 256 = 9216 dimensional. Ignoring boundary effects, each pool5 unit has a receptive field of 195×195 pixels in the original 227×227 pixel input. A central pool5 unit has a nearly global view, while one near the edge has a smaller, clipped support.

We visualize units from layer pool5, the max-pool output layer of the network's fifth and final convolutional layer. The pool5 feature map is 6 × 6 × 256 = 9216 dimensional. Ignoring boundary effects, each pool5 unit has a receptive field of 195 × 195 pixels in the original 227 × 227 pixel input. While one near the edge has small clip support, a central pool5 unit can see almost globally.

Each row in Figure 4 displays the top 16 activations for a pool5 unit from a CNN that we fine-tuned on VOC 2007 trainval. Six of the 256 functionally unique units are visualized (Appendix D includes more). These units were selected to show a representative sample of what the network learns. In the second row, we see a unit that fires on dog faces and dot arrays. The unit corresponding to the third row is a red blob detector. There are also detectors for human faces and more abstract patterns such as text and triangular structures with windows. The network appears to learn a representation that combines a small number of class-tuned features together with a distributed representation of shape, texture, color, and material properties. The subsequent fully connected layer fc6 has the ability to model a large set of compositions of these rich features.

Each row of the figure of the pool5 device from the CNN we fine-tuned on VOC 2007 trainval shows the top 4 (16) activations. 6 of the 256 functionally unique units (Appendix D includes more) are visualized. These devices were selected to show a representative sample of what the network learns. In the second row, see the device firing on dog faces and dot arrays. The part corresponding to the third row is a red blob detection part. There are also detectors for more abstract patterns with text and triangular structures in windows, such as human faces. The network appears to learn a representation that combines a small number of class-tuned features together with a distributed representation of shape, texture, color, and material properties. The subsequent fully connected layer FC6 has the ability to model a large set of compositions of these various features.

Performance layer-by-layer, without fine-tuning. To understand which layers are critical for detection performance, we analyzed results on the VOC 2007 dataset for each of the CNN’s last three layers. Layer pool5 was briefly described in Section 3.1. The final two layers are summarized below.

Performance layer by layer, without fine-tuning. To understand the layers important to detection performance, we analyzed VOC 2007 dataset results with each of the CNN's last 3 layers. Layer pool5 was briefly described in Section 3.1. The last two layers are summarized below.

Layer fc6 is fully connected to pool5 . To compute features, it multiplies a 4096×9216 weight matrix by the pool5 feature map (reshaped as a 9216-dimensional vector) and then adds a vector of biases. This intermediate vector is component-wise half-wave rectified (x ← max(0, x)).

Layer FC6 is fully connected to pool5. To compute features, it multiplies the pool5 feature map (reshaped as a 9216-dimensional vector) by a 4096 × 9216 weight matrix and adds a vector of biases. This intermediate vector has many components and is half-wave rectified (X ← max (0, x)).

Layer fc7 is the final layer of the network. It is implemented by multiplying the features computed by fc6 by a 4096 × 4096 weight matrix, and similarly adding a vector of biases and applying half-wave rectification.

Layer FC7 is the final layer of the network. It is implemented by multiplying the features computed by FC6 by a 4096 × 4096 weight matrix and, likewise, adding a vector of biases and applying half-wave rectification.

We start by looking at results from the CNN without fine-tuning on PASCAL, i.e. all CNN parameters were pre-trained on ILSVRC 2012 only. Analyzing performance layer-by-layer (Table 2 rows 1-3) reveals that features from fc7 generalize worse than features from fc6. This means that 29%, or about 16.8 million, of the CNN’s parameters can be removed without degrading mAP. More surprising is that removing both fc7 and fc6 produces quite good results even though pool5 features are computed using only 6% of the CNN’s parameters. Much of the CNN’s representational power comes from its convolutional layers, rather than from the much larger densely connected layers. This finding suggests potential utility in computing a dense feature map, in the sense of HOG, of an arbitrary-sized image by using only the convolutional layers of the CNN. This representation would enable experimentation with sliding-window detectors, including DPM, on top of pool5 features.

We start by looking at CNN results without fine-tuning on Pascal, i.e., all CNN parameters were pre-trained on ILSVRC 2012 only. Analyzing performance layer by layer (Table 2 rows 1-3) shows that features from FC7 generalize more than features from FC6. This means that 29% of the CNN's parameters, or about 16.8, can be removed without degrading the map. More surprising is that removing both FC7 and FC6 yields quite good results even though pool5 features are computed using 6% of the CNN's parameters. Most of the CNN's representational power comes from the convolutional layers rather than from the much more densely connected layers. This finding suggests the potential utility of computing a high-density feature map, in terms of pigs, of an arbitrary-sized image using the convolutional layers of the CNN. This representation enables experiments with sliding-window detectors, including DPM, on top of pool5 features.

Performance layer-by-layer, with fine-tuning. We now look at results from our CNN after having fine-tuned its parameters on VOC 2007 trainval. The improvement is striking (Table 2 rows 4-6): fine-tuning increases mAP by 8.0 percentage points to 54.2%. The boost from fine-tuning is much larger for fc6 and fc7 than for pool5 , which suggests that the pool5 features learned from ImageNet are general and that most of the improvement is gained from learning domain-specific non-linear classifiers on top of them.

Performance layer by layer, with fine-tuning. We now look at our CNN results after having fine-tuned the parameters on VOC 2007 trainval. The improvement (Table 2 rows 4-6) is being illuminated: fine-tuning increases the map by 8.0% points to 54.2%. The boost from fine-tuning is much larger for FC6 and FC7 than for pool5, suggesting that the pool5 features learned from ImageNet are general and that most of the improvement comes from learning region-specific nonlinear classification on top of them.

Comparison to recent feature learning methods. Relatively few feature learning methods have been tried on PASCAL VOC detection. We look at two recent approaches that build on deformable part models. For reference, we also include results for the standard HOG-based DPM [20].

Comparing recent feature learning methods. Relatively few feature learning methods are attempting PASCAL VOC detection. We look at recent approaches to building deformable part models. For reference, we also include the results of the standard HOG-based DPM [20].

The first DPM feature learning method, DPM ST [28], augments HOG features with histograms of “sketch token” probabilities. Intuitively, a sketch token is a tight distribution of contours passing through the center of an image patch. Sketch token probabilities are computed at each pixel by a random forest that was trained to classify 35×35 pixel patches into one of 150 sketch tokens or background.

The first DPM, this method's learning feature, DPM ST [28] extends HOG features with histograms of "sketch token" probabilities. Intuitively, a sketch token is a tight distribution of contours passing through the center of an image patch. Sketch token probabilities are computed at each pixel of a random forest trained to classify 35 × 35 pixel patches into one of 150 sketch tokens or background.

The second method, DPM HSC [31], replaces HOG with histograms of sparse codes (HSC). To compute an HSC, sparse code activations are solved for at each pixel using a learned dictionary of 100 7 × 7 pixel (grayscale) atoms. The resulting activations are rectified in three ways (full and both half-waves), spatially pooled, unit `2 normalized, and then power transformed (x ← sign(x)|x|α).

The second method, DPM HSC [31], replaces HOG with histograms of sparse codes (HSC). To compute HSC, sparse code activations are solved for using a learned dictionary of 100 7 × 7 pixel (grayscale) atoms at each pixel. (| X | α X ← symbol (X)) The resulting activations are rectified in three ways (both full and half waves), spatially pooled, unit`2 normalized, and then power-supply transformed.

All R-CNN variants strongly outperform the three DPM baselines (Table 2 rows 8-10), including the two that use feature learning. Compared to the latest version of DPM, which uses only HOG features, our mAP is more than 20 percentage points higher: 54.2% vs. 33.7%—a 61% relative improvement. The combination of HOG and sketch tokens yields 2.5 mAP points over HOG alone, while HSC improves over HOG by 4 mAP points (when compared internally to their private DPM baselines—both use nonpublic implementations of DPM that underperform the open source version [20]). These methods achieve mAPs of 29.1% and 34.3%, respectively

All R-CNN variants strongly outperform the three DPM baselines (Table 2 rows 8-10), including the two that use feature learning. Compared to the latest version of DPM using only HOG features, our map is 20% points higher: 54.2% vs. 33.7% -a 61% relative improvement. While HSC improves over HOG by 4 map points, the combination of HOG and sketch tokens yields 2.5 map points over HOG alone (compared internally to their own private DPM baselines—both use nonpublic implementations of DPM, open-source version yields [20]). These methods achieve maps of 29.1% and 34.3%, respectively

Most results in this paper use the network architecture from Krizhevsky et al. [25]. However, we have found that the choice of architecture has a large effect on R-CNN detection performance. In Table 3 we show results on VOC 2007 test using the 16-layer deep network recently proposed by Simonyan and Zisserman [43]. This network was one of the top performers in the recent ILSVRC 2014 classification challenge. The network has a homogeneous structure consisting of 13 layers of 3 × 3 convolution kernels, with five max pooling layers interspersed, and topped with three fully-connected layers. We refer to this network as “O-Net” for OxfordNet and the baseline as “T-Net” for TorontoNet.

Most results in this paper use the network architecture from Krizhevsky et al. [25]. However, we found that the choice of architecture has a large effect on R-CNN detection performance. In Table 3, we show results on the volatile organic compounds (VOC) 2007 test using the 16-layer deep network recently proposed by Simonyan and Zisserman [43]. This network was one of the best performances in the recent ILSVRC 2014 classification challenge. The network has a uniform structure consisting of 13 layers of 3 × 3 convolution kernels with five max pooling layers interspersed, and occupied three fully connected layers. We refer to this network as "O-Net" for OxfordNet and, as the baseline, "T-Internet" for Toronto.

To use O-Net in R-CNN, we downloaded the publicly available pre-trained network weights for the VGG ILSVRC 16 layers model from the Caffe Model Zoo.1 We then fine-tuned the network using the same protocol as we used for T-Net. The only difference was to use smaller minibatches (24 examples) as required in order to fit within GPU memory. The results in Table 3 show that RCNN with O-Net substantially outperforms R-CNN with TNet, increasing mAP from 58.5% to 66.0%. However there is a considerable drawback in terms of compute time, with the forward pass of O-Net taking roughly 7 times longer than T-Net.

To use O-Internet in R-CNN, we downloaded publicly available pre-trained network weights for the VGG ILSVRC 16-layer model from CAFFE Model Zoo.1, and we then fine-tuned the network using the same protocol we used for T -Net. The only difference was using smaller small batches (e.g., 24) as required to fit within GPU memory. The results in the table with O-Net show RCNN substantially outperforms R-CNN with TNet, increasing the map from 58.5% to 66.0%, 3. However, there is a considerable drawback in terms of compute time, the forward pass of O-Net, taking about 7 hours longer than T-Internet.

We applied the excellent detection analysis tool from Hoiem et al. [23] in order to reveal our method’s error modes, understand how fine-tuning changes them, and to see how our error types compare with DPM. A full summary of the analysis tool is beyond the scope of this paper and we encourage readers to consult [23] to understand some finer details (such as “normalized AP”). Since the analysis is best absorbed in the context of the associated plots, we present the discussion within the captions of Figure 5 and Figure 6.

We applied the excellent detection analysis tool from Hoiem et al. [23] in order to reveal our method's error modes, understand how fine-tuning changes them, and see how our error types compare with DPM. A full summary of the analysis tool is beyond the scope of this document, and we encourage readers to consult [23] to understand some finer details (such as "standardized AP"). Because the analysis is best absorbed in the context of the related plots, we present the discussion within the subtitles of Figure 5 and Figure 6.

Based on the error analysis, we implemented a simple method to reduce localization errors. Inspired by the bounding-box regression employed in DPM [17], we train a linear regression model to predict a new detection window given the pool5 features for a selective search region proposal. Full details are given in Appendix C. Results in Table 1, Table 2, and Figure 5 show that this simple approach fixes a large number of mislocalized detections, boosting mAP by 3 to 4 points.

Based on the error analysis, we implemented a simple method to reduce grasping errors. Inspired by the bounding box regression used in DPM [17], we train a linear regression model that predicts a new search window given the pool5 features for a selective search region proposal. Details boosting the map by 3 to 4 points, Table 1, Table 2, the simple approach fixing a large number of mislocalized detections, Figure 5 is given in Appendix C. Results.

Qualitative detection results on ILSVRC2013 are presented in Figure 8 and Figure 9 at the end of the paper. Each image was sampled randomly from the val2 set and all detections from all detectors with a precision greater than 0.5 are shown. Note that these are not curated and give a realistic impression of the detectors in action. More qualitative results are presented in Figure 10 and Figure 11, but these have been curated. We selected each image because it contained interesting, surprising, or amusing results. Here, also, all detections at precision greater than 0.5 are shown

Qualitative detection results of ILSVRC2013 are shown in Figure 8 and Figure 9 at the end of the sheet of paper. Each image is randomly sampled from the val2 set, and all detections from all detectors with precision greater than 0.5 are shown. Take care that these are not curated, and they give a realistic feel of the detectors in action. More qualitative results are presented in Figure 10 and Figure 11, but these have been curated. We selected each image because it contains fun, surprising, or amusing results. Here, also, all detections with precision greater than 0.5 are shown