Fast R-CNN:Paper

This paper proposes a Fast Region-based Convolutional Network method (Fast R-CNN) for object detection. Fast R-CNN builds on previous work to efficiently classify object proposals using deep convolutional networks. Compared to previous work, Fast R-CNN employs several innovations to improve training and testing speed while also increasing detection accuracy. Fast R-CNN trains the very deep VGG16 network 9× faster than R-CNN, is 213× faster at test-time, and achieves a higher mAP on PASCAL VOC 2012. Compared to SPPnet, Fast R-CNN trains VGG16 3× faster, tests 10× faster, and is more accurate. Fast R-CNN is implemented in Python and C++ (using Caffe) and is available under the open-source MIT License at https://github.com/rbgirshick/fast-rcnn

In this paper, we propose a fast region-based convolutional network method (high-speed R-CNN) for object detection. Fast R-CNN builds on prior work to efficiently classify proposals of objects using deep convolution networks. Compared with previous research, fast R-CNN uses several innovations that improve training and test speed while increasing detection accuracy. Fast R-CNN trains the very deep VGG16 network that goes 9 × faster than R-CNN and is 213 × faster at test time, and achieves a higher map on PASCAL VOC 2012 compared with SPPnet; fast R-CNN is fast VGG16 3 × trains, 10 × faster tests, and is more accurate. Fast R-CNN is implemented in ++ Python and C (using CAFFE) and is available under the open-source MIT license at https://github.com/rbgirshick/fast-rcnn.

Recently, deep ConvNets [14, 16] have significantly improved image classification [14] and object detection [9, 19] accuracy. Compared to image classification, object detection is a more challenging task that requires more complex methods to solve. Due to this complexity, current approaches (e.g., [9, 11, 19, 25]) train models in multi-stage pipelines that are slow and inelegant.

Recently deep ConvNets [14, 16] greatly improved the accuracy of image classification [14] and object detection [9, 19]. Compared with image classification, object detection is a more challenging task that requires more complex methods than solving. Due to this complexity, current approaches (e.g., [9, 11, 19, 25]) are slow and elegant multi-stage pipe train models.

Complexity arises because detection requires the accurate localization of objects, creating two primary challenges. First, numerous candidate object locations (often called “proposals”) must be processed. Second, these candidates provide only rough localization that must be refined to achieve precise localization. Solutions to these problems often compromise speed, accuracy, or simplicity.

Because accurate localization of detected objects is required, complexity arises, creating two major challenges. First, numerous candidate object locations (often so-called "proposals") are processed. Second, these candidates provide only rough localization that must be refined to achieve accurate localization. Solutions to these problems often damage speed, accuracy, and simplicity.

In this paper, we streamline the training process for stateof-the-art ConvNet-based object detectors [9, 11]. We propose a single-stage training algorithm that jointly learns to classify object proposals and refine their spatial locations.

In this paper, we simplify the training process for stateof state-of-the-art ConvNet-based object detectors [9, 11]. We propose a single-stage training algorithm that learns to classify joint-purpose proposals and correct their own spatial locations.

The resulting method can train a very deep detection network (VGG16 [20]) 9× faster than R-CNN [9] and 3× faster than SPPnet [11]. At runtime, the detection network processes images in 0.3s (excluding object proposal time) while achieving top accuracy on PASCAL VOC 2012 [7] with a mAP of 66% (vs. 62% for R-CNN).1

The resulting method can train a very deep detection network (VGG16 [20]) that is 9 × faster than R-CNN that is [9] 3 × faster than SPPnet [11]. At runtime, the detection network achieves top accuracy on PASCAL VOC 2012 [7] while a map of 66 % (vs. R-CNN 62 %) .1 processes images of 0.3S (excluding object proposal time)

The Region-based Convolutional Network method (RCNN) [9] achieves excellent object detection accuracy by using a deep ConvNet to classify object proposals. R-CNN, however, has notable drawbacks:

The region-based convolutional network method (RCNN) [9] achieves excellent object detection accuracy using a deep ConvNet for object proposal classification. R-CNN, however, has notable drawbacks:

1. Training is a multi-stage pipeline. R-CNN first finetunes a ConvNet on object proposals using log loss. Then, it fits SVMs to ConvNet features. These SVMs act as object detectors, replacing the softmax classi- fier learnt by fine-tuning. In the third training stage, bounding-box regressors are learned.

1. Training is a multi-stage pipeline. R-CNN first finetunes a ConvNet on proposals of objects using log loss. Then, it fits SVMs to ConvNet functions. These SVMs act as object detectors, replacing the learning by fine-tuning of softmax to classi- Fier. In the third training stage, bounding box regression is learned.

2. Training is expensive in space and time. For SVM and bounding-box regressor training, features are extracted from each object proposal in each image and written to disk. With very deep networks, such as VGG16, this process takes 2.5 GPU-days for the 5k images of the VOC07 trainval set. These features require hundreds of gigabytes of storage.

2. Training is expensive in time and space. For SVM and bounding box regression training, functions are extracted from the proposal of each object in each image and recorded to disk. With very deep networks such as VGG16, this process takes 2.5 GPU-days for the 5K images of the VOC07 trainval set. These functions require hundreds of gigabytes of storage.

3. Object detection is slow. At test-time, features are extracted from each object proposal in each test image. Detection with VGG16 takes 47s / image (on a GPU).

3. Object detection is slow. At test time, functions are extracted from the proposal of each object in each test image. Detection with VGG16 (on a GPU) does 47s / image.

R-CNN is slow because it performs a ConvNet forward pass for each object proposal, without sharing computation. Spatial pyramid pooling networks (SPPnets) [11] were proposed to speed up R-CNN by sharing computation. The SPPnet method computes a convolutional feature map for the entire input image and then classifies each object proposal using a feature vector extracted from the shared feature map. Features are extracted for a proposal by maxpooling the portion of the feature map inside the proposal into a fixed-size output (e.g., 6 × 6). Multiple output sizes are pooled and then concatenated as in spatial pyramid pooling [15]. SPPnet accelerates R-CNN by 10 to 100× at test time. Training time is also reduced by 3× due to faster proposal feature extraction.

R-CNN is slow because it performs a ConvNet forward pass for the proposal of each object without sharing this computation. Spatial pyramid pooling networks (SPPnets) [11] were proposed to shorten computation R-CNN by sharing. The SPPnet method computes a convolutional feature map for the entire input image, then classifies the proposal for each object using a feature vector extracted from the shared function map. Features extract proposals by maxpooling the function map portion inside the proposal into a fixed-size output (e.g., 6 × 6). Multiple output sizes are pooled and concatenated as in spatial pyramid pooling [15]. SPPnet accelerates R-CNN by 10 ~ 100 × at test time. Training time is also reduced by × 3 due to fast proposal feature extraction.

SPPnet also has notable drawbacks. Like R-CNN, training is a multi-stage pipeline that involves extracting features, fine-tuning a network with log loss, training SVMs, and finally fitting bounding-box regressors. Features are also written to disk. But unlike R-CNN, the fine-tuning algorithm proposed in [11] cannot update the convolutional layers that precede the spatial pyramid pooling. Unsurprisingly, this limitation (fixed convolutional layers) limits the accuracy of very deep networks.

SPPnet also has notable drawbacks. Like R-CNN, training is a multi-stage pipeline including feature extraction, and fine-tunes the network with log reduction, training SVMs, and finally fitting bounding box regression. Functions are recorded to disk. But unlike R-CNN, the fine-tuning algorithm cannot update the convolutional layers before spatial pyramid pooling, proposed in [11]. Naturally this limitation (fixed convolutional layers) limits the accuracy of very deep networks.

We propose a new training algorithm that fixes the disadvantages of R-CNN and SPPnet, while improving on their speed and accuracy. We call this method Fast R-CNN because it’s comparatively fast to train and test. The Fast RCNN method has several advantages:

While improving their own speed and accuracy, we propose a new training algorithm that solves the drawbacks of R-CNN and SPPnet. Because it is relatively fast to train and test, we, please contact fast R-CNN about this method. The fast RCNN method has several advantages:

1. Higher detection quality (mAP) than R-CNN, SPPnet

1. High detection quality (map) than R-CNN SPPnet

2. Training is single-stage, using a multi-task loss

2. Training, using multi-task loss, which is single-stage

3. Training can update all network layers

3. Training to update all network layers

4. No disk storage is required for feature caching

4. Any disk storage does not require function caching

Fast R-CNN is written in Python and C++ (Caffe [13]) and is available under the open-source MIT License at https://github.com/rbgirshick/fast-rcnn

Fast R-CNN is written in Python and C ++ (CAFFE [13]) and is available under the open-source MIT license at https://github.com/rbgirshick/fast-rcnn.

Fig. 1 illustrates the Fast R-CNN architecture. A Fast R-CNN network takes as input an entire image and a set of object proposals. The network first processes the whole image with several convolutional (conv) and max pooling layers to produce a conv feature map. Then, for each object proposal a region of interest (RoI) pooling layer extracts a fixed-length feature vector from the feature map. Each feature vector is fed into a sequence of fully connected (fc) layers that finally branch into two sibling output layers: one that produces softmax probability estimates over K object classes plus a catch-all “background” class and another layer that outputs four real-valued numbers for each of the K object classes. Each set of 4 values encodes refined bounding-box positions for one of the K classes.

Fig. FIG. 1 shows the high-speed R-CNN architecture. A fast R-CNN network takes as input the entire image and a set of object proposals. The network first processes the entire image with several convolutional (conversion) and max pooling layers to generate a conversion function map. Then, for each object proposal, the region of the interest (ROI) pooling layer extracts a fixed-length feature vector from the function map. The one that generated softmax probability estimates over K object classes plus a catch-all "background" class and another layer outputting real four: each feature vector is fed into a sequence of fully connected (FC) layers that finally branch into two sibling output layers -valued numbers for each K object class. Each set of 4 values encodes the position of a refined bounding box for one of the K classes.

The RoI pooling layer uses max pooling to convert the features inside any valid region of interest into a small feature map with a fixed spatial extent of H × W (e.g., 7 × 7), where H and W are layer hyper-parameters that are independent of any particular RoI. In this paper, an RoI is a rectangular window into a conv feature map. Each RoI is defined by a four-tuple (r, c, h, w) that specifies its top-left corner (r, c) and its height and width (h, w)

The return on investment (ROI) pooling layer uses max pooling to convert the features inside a valid region of interest into a small function map with a fixed spatial extent of W × H (e.g., 7 × 7); H and W are layer hyper parameters, independent of a particular RoI. In this paper, return on investment is a rectangular window into the conversion function map. Each RoI is defined by a four-tuple (w R, C, H,) that specifies its top-left corner (R, C) and height (w hours) width

RoI max pooling works by dividing the h × w RoI window into an H × W grid of sub-windows of approximate size h/H × w/W and then max-pooling the values in each sub-window into the corresponding output grid cell. Pooling is applied independently to each feature map channel, as in standard max pooling. The RoI layer is simply the special-case of the spatial pyramid pooling layer used in SPPnets [11] in which there is only one pyramid level. We use the pooling sub-window calculation given in [11].

RoI max / w × reference size H / h sub-window W grid × H divides the RoI window W × H works pooling, then W and max-pooling the values of each sub-window into the corresponding output grid cell. Pooling is applied independently to each function map channel, as in standard max pooling. The region of interest layer is simply a special case of the spatial pyramid pooling layer used in SPPnets [11], where there is only one pyramid level. We use the pooling sub-window calculation given in [11].

We experiment with three pre-trained ImageNet [4] networks, each with five max pooling layers and between five and thirteen conv layers (see Section 4.1 for network details). When a pre-trained network initializes a Fast R-CNN network, it undergoes three transformations.

We, with five max pooling layers, experiment with each of three pre-trained ImageNet [4] networks, and between five thirteen conversion layers (see Section 4.1 for network details). When a pre-trained network initializes a high-speed R-CNN network, it undergoes three transformations.

First, the last max pooling layer is replaced by a RoI pooling layer that is configured by setting H and W to be compatible with the net’s first fully connected layer (e.g., H = W = 7 for VGG16).

First, the recent max pooling layer (e.g., H = W = 7 for VGG16) is replaced by a RoI pooling layer configured by setting H and W to be compatible with the net first fully connected layer.

Second, the network’s last fully connected layer and softmax (which were trained for 1000-way ImageNet classifi- cation) are replaced with the two sibling layers described earlier (a fully connected layer and softmax over K + 1 categories and category-specific bounding-box regressors).

Second, through K + 1 categories and category-specific bounding- earlier (the network's last fully connected layer and softmax (trained for 1000-way ImageNet, separator terminal) are replaced with the two sibling layers that describe the fully connected layer and softmax box regression).

Third, the network is modified to take two data inputs: a list of images and a list of RoIs in those images.

List of images and list of ROIs in those images: Third, the network is modified to use two data inputs.

Training all network weights with back-propagation is an important capability of Fast R-CNN. First, let’s elucidate why SPPnet is unable to update weights below the spatial pyramid pooling layer.

Training all network weights by back propagation is an important function of fast R-CNN. First, the reason SPPnet cannot update weights below the spatial pyramid pooling layer can be made clear.

The root cause is that back-propagation through the SPP layer is highly inefficient when each training sample (i.e. RoI) comes from a different image, which is exactly how R-CNN and SPPnet networks are trained. The inefficiency stems from the fact that each RoI may have a very large receptive field, often spanning the entire input image. Since the forward pass must process the entire receptive field, the training inputs are large (often the entire image).

The root cause is that back propagation through the SPP layer is very inefficient when each training sample (i.e., return on investment) comes from a different image, exactly how R-CNN and SPPnet networks are trained. The inefficiency stems from the fact that each return on investment can have a very large receptive field, often spanning the entire input image. Because the forward pass must process the entire receptive field, the training input is large (usually the entire image).

We propose a more efficient training method that takes advantage of feature sharing during training. In Fast RCNN training, stochastic gradient descent (SGD) minibatches are sampled hierarchically, first by sampling N images and then by sampling R/N RoIs from each image. Critically, RoIs from the same image share computation and memory in the forward and backward passes. Making N small decreases mini-batch computation. For example, when using N = 2 and R = 128, the proposed training scheme is roughly 64× faster than sampling one RoI from 128 different images (i.e., the R-CNN and SPPnet strategy).

We propose a more efficient training method that uses shared functions during training. In fast RCNN training, stochastic gradation descent (SGD) minibatches are sampled hierarchically, first by sampling N images and then by sampling R / N ROIs from each image. Critically, ROIs from the same image share computation and memory in the forward and backward passes. N making small reduces mini batch computation. For example, when using N = 2 and R = 128, the proposed training scheme is about 64 × speed than sampling one ROI from different images (128) (i.e., R-CNN and SPPnet strategy).

One concern over this strategy is it may cause slow training convergence because RoIs from the same image are correlated. This concern does not appear to be a practical issue and we achieve good results with N = 2 and R = 128 using fewer SGD iterations than R-CNN.

One concern through this strategy is that Roas from the same image are correlated, so it can cause slow training convergence. These problems do not appear as practical problems; we achieve good results using fewer SGD iterations than N = 2 and R = 128 R-CNN.

In addition to hierarchical sampling, Fast R-CNN uses a streamlined training process with one fine-tuning stage that jointly optimizes a softmax classifier and bounding-box regressors, rather than training a softmax classifier, SVMs, and regressors in three separate stages [9, 11]. The components of this procedure (the loss, mini-batch sampling strategy, back-propagation through RoI pooling layers, and SGD hyper-parameters) are described below.

In addition to hierarchical sampling, fast R-CNN, rather than training a softmax classifier and SVMs jointly with one fine-tuning, uses a simplified training process of a stage that optimizes the softmax classifier and bounding box regression variables, and regression in 3 separate stages [9 11]. The components of this procedure (loss through ROI pooling layers, mini batch sampling method, back propagation and SGD hyper parameters) are described.

Multi-task loss. A Fast R-CNN network has two sibling output layers. The first outputs a discrete probability distribution (per RoI), p = (p0, . . . , pK), over K + 1 categories. As usual, p is computed by a softmax over the K+1 outputs of a fully connected layer. The second sibling layer outputs bounding-box regression offsets, t^k = t_x^k, t_y^k, t_w^k, t_h^k), for each of the K object classes, indexed by k. We use the parameterization for t^k given in [9], in which t^k specifies a scale-invariant translation and log-space height/width shift relative to an object proposal.

Multiple task loss. A fast R-CNN network holds two sibling output layers. The first is a (per ROI) discrete probability distribution, P = output (P0... pharmacokinetics), K + 1 kinds or more. As usual, the page computes softmax over the K + 1 outputs of the fully connected layer. The output bounding box regression offset of the second sibling layer, t ^ K = t_x ^ K, t_y ^ K, t_w ^ K, t_h ^ K), for each of the K object classes indexed by K. We use the parameters for t ^ k given in [9], where t ^ k specifies a scale-invariant translation and a log-space height / width shift relative to the object's proposal.

Each training RoI is labeled with a ground-truth class u and a ground-truth bounding-box regression target v. We use a multi-task loss L on each labeled RoI to jointly train for classification and bounding-box regression:

. Each training return on investment (ROI) is marked with a ground truth class u and a ground truth bounding box regression target v, and we use a multiple task loss L on each labeled RoI to jointly train classification and bounding box regression

(1)

in which Lcls(p, u) = − log pu is log loss for true class u. The second task loss, Lloc, is defined over a tuple of true bounding-box regression targets for class u, v = (vx, vy, vw, vh), and a predicted tuple t^u = t_x^u, t_y^u, t_w^u, t_h^u), again for class u. The Iverson bracket indicator function [u ≥ 1] evaluates to 1 when u ≥ 1 and 0 otherwise. By convention the catch-all background class is labeled u = 0. For background RoIs there is no notion of a ground-truth bounding box and hence Lloc is ignored. For bounding-box regression, we use the loss

Which LCLS (P, U) = - log urethane is the log loss for the true class U. The second task loss, Lloc, is defined over a tuple of true bounding box regression targets for class U, V = (VX, VY, Volkswagen, VH), and the predicted tuple t ^ U = t_x ^ yu, t_y ^ U, t_w ^ U, t_h ^ U) again for class yu. The Iverson bracket display function evaluates [U ≥ 1] to 1 when yu ≥ 1, 0 otherwise. By convention, the catch-all background class u is labeled 0 = for background ROIs, where there is no concept of this ground truth bounding box and hence Lloc is ignored. For bounding box regression, we use the loss

(2)

in which

which

(3)

is a robust L1 loss that is less sensitive to outliers than the L2 loss used in R-CNN and SPPnet. When the regression targets are unbounded, training with L2 loss can require careful tuning of learning rates in order to prevent exploding gradients. Eq. 3 eliminates this sensitivity.

It is an L1 robust reduction that is less sensitive to outliers than the L2 loss used in R-CNN and SPPnet. When the regression target is bound, L2 loss training may require careful adjustment of learning ratios to prevent explosion gradients. Eq. 3 removes this sensitivity.

The hyper-parameter λ in Eq. 1 controls the balance between the two task losses. We normalize the ground-truth regression targets vi to have zero mean and unit variance. All experiments use λ = 1.

The hyper parameter λ of Eq. Controls the balance of 1 task loss. We normalize the ground truth regression target VI to have zero mean and unit variance. All experiments use λ 1 =.

We note that [6] uses a related loss to train a classagnostic object proposal network. Different from our approach, [6] advocates for a two-network system that separates localization and classification. OverFeat [19], R-CNN [9], and SPPnet [11] also train classifiers and bounding-box localizers, however these methods use stage-wise training, which we show is suboptimal for Fast R-CNN (Section 5.1).

We have [6] using a related loss to train a classagnostic object proposal network. Unlike our approach, [6] advocates for a two-network system that separates localization and classification. OverFeat [19], R-CNN [9] and SPPnet [11] also, but these methods use stage-smart training, which we show is optimal for fast R-CNN (Section 5.1), and cultivate classification and bounding box localization.

Mini-batch sampling. During fine-tuning, each SGD mini-batch is constructed from N = 2 images, chosen uniformly at random (as is common practice, we actually iterate over permutations of the dataset). We use mini-batches of size R = 128, sampling 64 RoIs from each image. As in [9], we take 25% of the RoIs from object proposals that have intersection over union (IoU) overlap with a groundtruth bounding box of at least 0.5. These RoIs comprise the examples labeled with a foreground object class, i.e. u ≥ 1. The remaining RoIs are sampled from object proposals that have a maximum IoU with ground truth in the interval [0.1, 0.5), following [11]. These are the background examples and are labeled with u = 0. The lower threshold of 0.1 appears to act as a heuristic for hard example mining [8]. During training, images are horizontally flipped with probability 0.5. No other data augmentation is used.

Mini batch sampling. During fine-tuning, each SGD mini batch consists of N = 2 images chosen arbitrarily (because it is common practice, we actually iterate over permutations of the dataset). We use mini batches of size R = 128, sampling 64 ROIs from each image. As in [9], take 25 % of Roas from object proposals at the intersection through the union (IOU) overlap of groundtruth bounding boxes of at least 0.5. These Roas, i.e. yu ≥ 1. include examples labeled with a foreground object class, [11] following the remaining Roas being sampled from object proposals with a maximum IOU at interval [0.1 ground truth, 0.5). These background examples 0.1 appears 0.1 lower threshold acts as a heuristic for mining hard examples, marked as = U [8]. During training, images are flipped horizontally with probability 0.5. No other data increase is used.

Back-propagation through RoI pooling layers. Backpropagation routes derivatives through the RoI pooling layer. For clarity, we assume only one image per mini-batch (N = 1), though the extension to N > 1 is straightforward because the forward pass treats all images independently.

Re-propagation through RoI pooling layers. Back propagation path derivative products through the return on investment (ROI) pooling layer. Extension to N> 1 is simple because the forward pass treats all images independently, but for clarity, we assume one image per mini batch (N = 1).

Let xi ∈ R be the i-th activation input into the RoI pooling layer and let yrj be the layer’s j-th output from the rth RoI. The RoI pooling layer computes y_rj = x_i∗(r,j), in which i∗(r, j) = argmax_i0∈R(r,j) xi0 . R(r, j) is the index set of inputs in the sub-window over which the output unit yrj max pools. A single xi may be assigned to several different outputs yrj .

Let XI ∈ R be the i-th activation input into the return on investment (ROI) pooling layer, and let yrj be the j-th output of the layer from the r-th return on investment (ROI). The return on investment (ROI) pooling layer computes y_rj from = x_i * (R, J), where I (R, J) = argmax_i0∈R (R, J) xi0 *. R (R, j) is the input set index into output unit yrj max pool through the sub window. A single XI can be assigned to several outputs yrj.

The RoI pooling layer’s backwards function computes partial derivative of the loss function with respect to each input variable xi by following the argmax switches:

The lower function of the ROI pooling layer computes the partial derivative of the loss function with respect to each input variable XI, following the argmax switches:

(4)

In words, for each mini-batch RoI r and for each pooling output unit yrj , the partial derivative ∂L/∂yrj is accumulated if i is the argmax selected for yrj by max pooling. In back-propagation, the partial derivatives ∂L/∂yrj are already computed by the backwards function of the layer on top of the RoI pooling layer.

That is, for each pooling output unit yrj for each mini batch return on investment (ROI) r, the partial derivative ∂L / ∂yrj is accumulated if I selected argmax for yrj by max pooling. In back propagation, the partial derivative ∂L / ∂yrj is already computed by the lower function of the layer on top of the return on investment (ROI) pooling layer.

SGD hyper-parameters. The fully connected layers used for softmax classification and bounding-box regression are initialized from zero-mean Gaussian distributions with standard deviations 0.01 and 0.001, respectively. Biases are initialized to 0. All layers use a per-layer learning rate of 1 for weights and 2 for biases and a global learning rate of 0.001. When training on VOC07 or VOC12 trainval we run SGD for 30k mini-batch iterations, and then lower the learning rate to 0.0001 and train for another 10k iterations. When we train on larger datasets, we run SGD for more iterations, as described later. A momentum of 0.9 and parameter decay of 0.0005 (on weights and biases) are used.

Set the SGD hyper parameters. The fully connected layers used for softmax classification and bounding box regression analysis are initialized with zero-mean Gaussian distributions with standard deviations of 0.01 and 0.001, respectively. Biases are initialized to 0, using a rate of learning layer weight (1) per all layers, 2 bias and a 0.001 global learning rate. When training on VOC07 or VOC12 trainval, we run SGD for 30K mini batch iterations and lower the rate and the train to learning 0.0001 of another 10,000 iterations. As described later, when we train on large datasets, we run SGD for more iterations. A momentum of 0.0005 0.9 and rot for parameters (on weight and prejudice) is used.

We explore two ways of achieving scale invariant object detection: (1) via “brute force” learning and (2) by using image pyramids. These strategies follow the two approaches in [11]. In the brute-force approach, each image is processed at a pre-defined pixel size during both training and testing. The network must directly learn scale-invariant object detection from the training data.

We explore two ways to achieve scale-invariant object detection: (1) through "armed force" learning, (2) using image pyramids. These strategies, please follow the two methods of [11]. In the brute-force approach, each image is processed at a predefined pixel size during both training and testing. The network must learn scale-invariant object detection directly from training data.

The multi-scale approach, in contrast, provides approximate scale-invariance to the network through an image pyramid. At test-time, the image pyramid is used to approximately scale-normalize each object proposal. During multi-scale training, we randomly sample a pyramid scale each time an image is sampled, following [11], as a form of data augmentation. We experiment with multi-scale training for smaller networks only, due to GPU memory limits.

The multi - scale method, in contrast, provides approximate scale invariance to the network through an image pyramid. At test, the image pyramid normalizes extending the proposal of each object used for about. During multi - scale training, each time we sample an image at pyramid scale, in the form of random data increase, following [11], sample. We experiment with multi-scale training of only small networks, due to GPU memory limits.

Once a Fast R-CNN network is fine-tuned, detection amounts to little more than running a forward pass (assuming object proposals are pre-computed). The network takes as input an image (or an image pyramid, encoded as a list of images) and a list of R object proposals to score. At test-time, R is typically around 2000, although we will consider cases in which it is larger (≈ 45k). When using an image pyramid, each RoI is assigned to the scale such that the scaled RoI is closest to 2242 pixels in area [11].

Once the high-speed R-CNN network becomes, to a little more than running a forward pass, fine-tuning of the detection amount (assuming object proposals are precomputed). The network takes as input an image (or an image pyramid, encoded as a list of images) and a list of R object proposals for scoring. We will consider larger cases (≈ 45K), but at test, R is generally about 2000. When using an image pyramid, each is assigned to the scale so that the scaled RoI is the pixel closest to 2242 in the return on investment area [11].

For each test RoI r, the forward pass outputs a class posterior probability distribution p and a set of predicted bounding-box offsets relative to r (each of the K classes gets its own refined bounding-box prediction). We assign a detection confidence to r for each object class k using the estimated probability Pr(class = k | r) ∆= pk. We then perform non-maximum suppression independently for each class using the algorithm and settings from R-CNN [9].

For each test return on investment R, the forward pass outputs a class posterior probability distribution P and a set of predicted bounding box offsets for R (each of the K classes gets its own refined bounding box prediction). (| R class = k) is Δ = PK; we assign detection confidence to research for each object class k using the estimated probability sleep. We then perform non-maximum suppression independently for each class using the algorithm and settings from R-CNN [9].

For whole-image classification, the time spent computing the fully connected layers is small compared to the conv layers. On the contrary, for detection the number of RoIs to process is large and nearly half of the forward pass time is spent computing the fully connected layers (see Fig. 2). Large fully connected layers are easily accelerated by compressing them with truncated SVD [5, 23].

The time for whole-image classification spent is small compared to the conversion layers computing the fully connected layers. Conversely, in detection (see FIG. 2) the number of process Roas is large, and nearly half of the forward pass time is spent computing the fully connected layers. Large fully connected layers are easily accelerated by compression with cut SVD [5, 23].

In this technique, a layer parameterized by the u × v weight matrix W is approximately factorized as

In this technique, the layer parameterized by the V weight matrix W × U is factorized into about

(5)

using SVD. In this factorization, U is a u × t matrix comprising the first t left-singular vectors of W, Σt is a t × t diagonal matrix containing the top t singular values of W, and V is v × t matrix comprising the first t right-singular vectors of W. Truncated SVD reduces the parameter count from uv to t(u + v), which can be significant if t is much smaller than min(u, v). To compress a network, the single fully connected layer corresponding to W is replaced by two fully connected layers, without a non-linearity between them. The first of these layers uses the weight matrix Σ_t*V^T (and no biases) and the second uses U (with the original biases associated with W). This simple compression method gives good speedups when the number of RoIs is large.

Using SVD. In this factor, U is an AU × T matrix containing the first t left singular vectors of W, Σt is a × t diagonal matrix containing the upper t singular values of W, and V is a t matrix × V containing the first t right ones is W. The truncated SVD singular vector t reduces the parameter count from ultraviolet (U + V), which may be important if t is smaller than minutes (U, V). To compress the network, the one complete connection layer corresponding to W is replaced by two fully connected layers without non-linearity between them. The first of these layers uses the weight matrix Σ_t V * ^ T (with no bias), and the second uses U (with the original biases associated with W). When the number of Roas is large, the simple compression method can see a good speedup effect.

Three main results support this paper’s contributions:

Three main results support the contributions of this paper

1. State-of-the-art mAP on VOC07, 2010, and 2012

On VOC07, 2010 and 2012, 1 state-of-the-art map

2. Fast training and testing compared to R-CNN, SPPnet

2. Fast training and R-CNN comparison test, SPPnet

3. Fine-tuning conv layers in VGG16 improves mAP

VGG16 3. Conversion layers of fine-tuning improve the map

Our experiments use three pre-trained ImageNet models that are available online.2 The first is the CaffeNet (essentially AlexNet [14]) from R-CNN [9]. We alternatively refer to this CaffeNet as model S, for “small.” The second network is VGG CNN M 1024 from [3], which has the same depth as S, but is wider. We call this network model M, for “medium.” The final network is the very deep VGG16 model from [20]. Since this model is the largest, we call it model L. In this section, all experiments use single-scale training and testing (s = 600; see Section 5.2 for details).

The first is CaffeNet (essentially AlexNet [14]) from R-CNN [9] online.2 Our experiments use three available pre-trained ImageNet models. We alternatively refer to this CaffeNet as model S for "small." The second network is CNN M 1024 from VGG [3], which has the same depth as S but becomes wider. We call this network model M, "medium." The last network is the very deep VGG16 model from [20]. Because this model is the maximum, we, all experiments use single-scale training and testing, call it model L in this section (s = 600; see Section 5.2 for details).

On these datasets, we compare Fast R-CNN (FRCN, for short) against the top methods on the comp4 (outside data) track from the public leaderboard (Table 2, Table 3).3 For the NUS NIN c2000 and BabyLearning methods, there are no associated publications at this time and we could not find exact information on the ConvNet architectures used; they are variants of the Network-in-Network design [17]. All other methods are initialized from the same pre-trained VGG16 network.

On these datasets, we compare fast R-CNN (FRCN for short) against the method for the (outside data) track (Table 2, Table 3) 0.3 from the COMP4 public leader in the case of the top NUS NIN C2000 and the BabyLearning method; in this case there are no related posts, and we cannot find exact information about the ConvNet architecture used; they are variants of the network - in - network design [17]. All other methods are initialized from the same pre-trained VGG16 network.

Fast R-CNN achieves the top result on VOC12 with a mAP of 65.7% (and 68.4% with extra data). It is also two orders of magnitude faster than the other methods, which are all based on the “slow” R-CNN pipeline. On VOC10, SegDeepM [25] achieves a higher mAP than Fast R-CNN (67.2% vs. 66.1%). SegDeepM is trained on VOC12 trainval plus segmentation annotations; it is designed to boost R-CNN accuracy by using a Markov random field to reason over R-CNN detections and segmentations from the O2P [1] semantic-segmentation method. Fast R-CNN can be swapped into SegDeepM in place of R-CNN, which may lead to better results. When using the enlarged 07++12 training set (see Table 2 caption), Fast R-CNN’s mAP increases to 68.8%, surpassing SegDeepM.

Fast R-CNN can get the top result on VOC12 with a map of 65.7 % (68.4 % with extra data). It is also two orders of magnitude, than other methods all based on the fast "slow" R-CNN pipeline. On VOC10, SegDeepM [25] achieves a higher map (67.2 % vs. 66.1 %) than fast R-CNN. SegDeepM trains on VOC12 trainval plus segmentation annotations; it is designed to increase R-CNN accuracy by using a Markov random field to reason over R-CNN detections and segmentation from the O2P [1] semantic - segmentation method. Fast R-CNN can be exchanged into SegDeepM in place of R-CNN, which can lead to better results. When using the enlarged 07 ++ (12) training set, surpassing SegDeepM, to 68.8 %, the map of fast R-CNN increases (see Table 2 caption).

On VOC07, we compare Fast R-CNN to R-CNN and SPPnet. All methods start from the same pre-trained VGG16 network and use bounding-box regression. The VGG16 SPPnet results were computed by the authors of [11]. SPPnet uses five scales during both training and testing. The improvement of Fast R-CNN over SPPnet illustrates that even though Fast R-CNN uses single-scale training and testing, fine-tuning the conv layers provides a large improvement in mAP (from 63.1% to 66.9%). R-CNN achieves a mAP of 66.0%. As a minor point, SPPnet was trained without examples marked as “difficult” in PASCAL. Removing these examples improves Fast R-CNN mAP to 68.1%. All other experiments use “difficult” examples.

On VOC07, we compare fast R-CNN to R-CNN and SPPnet. All methods start from the same pre-trained VGG16 network and use bounding box regression. The VGG16 SPPnet results were computed by the authors of [11]. SPPnet uses five weighing scales during both training and testing. The improvement of fast R-CNN over SPPnet shows that even though fast R-CNN uses single-scale training and testing, fine-tuning the conversion layers provides a large improvement in the map (from 63.1 % to 66.9 %). R-CNN can get a map of 66.0 %. As a small point, SPPnet trained without examples marked "difficult" in Pascal. Removing these examples improves the fast R-CNN map to 68.1 %. All other experiments use "difficult" examples.

Fast training and testing times are our second main result. Table 4 compares training time (hours), testing rate (seconds per image), and mAP on VOC07 between Fast RCNN, R-CNN, and SPPnet. For VGG16, Fast R-CNN processes images 146× faster than R-CNN without truncated SVD and 213× faster with it. Training time is reduced by 9×, from 84 hours to 9.5. Compared to SPPnet, Fast RCNN trains VGG16 2.7× faster (in 9.5 vs. 25.5 hours) and tests 7× faster without truncated SVD or 10× faster with it. Fast R-CNN also eliminates hundreds of gigabytes of disk storage, because it does not cache features.

Fast training and test time are our second week result. Table 4 compares training time (hours), tests speed (seconds per image), and the map on VOC07 between fast RCNN, R-CNN, and SPPnet. For VGG16, fast R-CNN processes images 146 × faster than R-CNN without fast and truncated SVD 213 ×. Training time is reduced by 9 × from 9.5 84 hours. Compared to SPPnet, fast RCNN this VGG16 2.7 × fast train and 7 × fast without SVD 10 × fast and test (9.5 vs. 25.5 hours). Because it fails to cache functions, fast R-CNN also removes hundreds of gigabytes of disk storage.

Truncated SVD. Truncated SVD can reduce detection time by more than 30% with only a small (0.3 percentage point) drop in mAP and without needing to perform additional fine-tuning after model compression. Fig. 2 illustrates how using the top 1024 singular values from the 25088 × 4096 matrix in VGG16’s fc6 layer and the top 256 singular values from the 4096×4096 fc7 layer reduces runtime with little loss in mAP. Further speed-ups are possible with smaller drops in mAP if one fine-tunes again after compression.

Truncated SVD. Truncated SVD can reduce detection time by more than 30 % with only a small (0.3 percentage point) drop in the map and without needing to perform additional fine-tuning after model compression. Fig. Shows how using the top 1,024 singular values from the 25088 × 4096 matrix in VGG16's FC6 layer and the top 256 singular values from the 4096 × 4096 FC7 layer reduces runtime with slight loss on the map. Also speed-up on the map with small droplets and, if possible, one fine-tuning again after compression.

For the less deep networks considered in the SPPnet paper [11], fine-tuning only the fully connected layers appeared to be sufficient for good accuracy. We hypothesized that this result would not hold for very deep networks. To validate that fine-tuning the conv layers is important for VGG16, we use Fast R-CNN to fine-tune, but freeze the thirteen conv layers so that only the fully connected layers learn. This ablation emulates single-scale SPPnet training and decreases mAP from 66.9% to 61.4% (Table 5). This experiment verifies our hypothesis: training through the RoI pooling layer is important for very deep nets

For the less deep networks considered in the SPPnet paper [11], fine-tuning only the fully connected layers was shown to be sufficient for good accuracy. We assumed this result would not hold very deep networks. To verify fine-tuning conversion layers is important to VGG16, we use fast R-CNN to - fine-tune, but freeze the thirteen conversion layers so that the fully connected layers can learn. This ablation emulates single-scale SPPnet training and reduces the map from 66.9 % to 61.4 % (Table 5). This experiment confirms our hypothesis: training through the return on investment (ROI) pooling layer is important for very deep nets

Does this mean that all conv layers should be fine-tuned? In short, no. In the smaller networks (S and M) we find that conv1 is generic and task independent (a well-known fact [14]). Allowing conv1 to learn, or not, has no meaningful effect on mAP. For VGG16, we found it only necessary to update layers from conv3 1 and up (9 of the 13 conv layers). This observation is pragmatic: (1) updating from conv2 1 slows training by 1.3× (12.5 vs. 9.5 hours) compared to learning from conv3 1; and (2) updating from conv1 1 over-runs GPU memory. The difference in mAP when learning from conv2 1 up was only +0.3 points (Table 5, last column). All Fast R-CNN results in this paper using VGG16 fine-tune layers conv3 1 and up; all experiments with models S and M fine-tune layers conv2 and up.

Does this mean all conversion layers should be fine-tuned? That is, no. In small networks (S and M), we find CONV1 is general and task independent (a well-known fact [14]). Letting CONV1 learn or not has no meaningful effect on the map. For VGG16, we only needed to update layers up from conv3 1 (9 of the 13 conversion layers). This observation is practical: (1) updating from CONV2 1 slows training by 1.3 × (12.5 vs. 9.5 hours) compared to learning from conv3 1; running over GPU memory and (2) updating from (1) CONV1. The difference in the map of learning from CONV2 1 was 0.3 points (Table 5, last column). All high-speed R-CNN results in this paper using VGG16 fine-tuning layers 1 up conv3; all experiments of models S and M fine-tuning layers CONV2 max.

We conducted experiments to understand how Fast RCNN compares to R-CNN and SPPnet, as well as to evaluate design decisions. Following best practices, we performed these experiments on the PASCAL VOC07 dataset.

We conducted experiments to evaluate design decisions as well as fast understanding of how RCNN compares R-CNN and SPPnet. Following best practices, we performed these experiments on the Pascal VOC07 dataset.

Multi-task training is convenient because it avoids managing a pipeline of sequentially-trained tasks. But it also has the potential to improve results because the tasks influence each other through a shared representation (the ConvNet) [2]. Does multi-task training improve object detection accuracy in Fast R-CNN?

Multiple task training is convenient because this can avoid managing a pipeline of sequentially trained tasks. It also has the potential to improve results because the tasks influence each other through a shared representation (ConvNet) [2]. Does multiple task training improve object detection precision in fast R-CNN?

To test this question, we train baseline networks that use only the classification loss, Lcls, in Eq. 1 (i.e., setting λ = 0). These baselines are printed for models S, M, and L in the first column of each group in Table 6. Note that these models do not have bounding-box regressors. Next (second column per group), we take networks that were trained with the multi-task loss (Eq. 1, λ = 1), but we disable boundingbox regression at test time. This isolates the networks’ classification accuracy and allows an apples-to-apples comparison with the baseline networks.

To test this question, we train baseline networks that use only the classification loss, LCLS, in Eq. 1 (i.e., set to λ = 0). These baselines are printed for models S, M, L and in the first column of each group in Table 6; note that this model has no bounding box regression variables. Next (second column per group), we take networks trained with the multiple task loss (Eq. 1, λ = 1), but we disable BoundingBox regression analysis at test. This isolates the classification accuracy of the networks and allows an apple-to-apple comparison with the baseline networks.

Across all three networks we observe that multi-task training improves pure classification accuracy relative to training for classification alone. The improvement ranges from +0.8 to +1.1 mAP points, showing a consistent positive effect from multi-task learning.

Across three networks, we observe that multiple task training improves the pure classification accuracy reference of training for classification alone. The improvement, showing a consistent positive effect from multiple task learning, +0.8 +1.1 mapping point range.

Finally, we take the baseline models (trained with only the classification loss), tack on the bounding-box regression layer, and train them with Lloc while keeping all other network parameters frozen. The third column in each group shows the results of this stage-wise training scheme: mAP improves over column one, but stage-wise training underperforms multi-task training (forth column per group).

Finally, we do baseline (trained with only classification loss) models on the bounding box regression layer, thumbtack, and train them with Lloc while keeping all other network parameters frozen. The third column of each group shows the results of this stage-smart training scheme: the map improves over column one, but stage-smart training underperforms multiple task training (etc. column per group).

We compare two strategies for achieving scale-invariant object detection: brute-force learning (single scale) and image pyramids (multi-scale). In either case, we define the scale s of an image to be the length of its shortest side.

Brute-force learning (single scale) and image pyramid (multi-scale): we compare two strategies for achieving scale-invariant object detection. In each case, we define the scale S of an image as the length of the short side.

All single-scale experiments use s = 600 pixels; s may be less than 600 for some images as we cap the longest image side at 1000 pixels and maintain the image’s aspect ratio. These values were selected so that VGG16 fits in GPU memory during fine-tuning. The smaller models are not memory bound and can benefit from larger values of s; however, optimizing s for each model is not our main concern. We note that PASCAL images are 384 × 473 pixels on average and thus the single-scale setting typically upsamples images by a factor of 1.6. The average effective stride at the RoI pooling layer is thus ≈ 10 pixels.

All single-scale experiments = 600 pixels are used; s may be 600 or more for some images less as we cap the longest image side at 1000 pixels, and can maintain the aspect ratio of the image. These values were selected so that VGG16 fits in GPU memory during fine-tuning. Smaller models are not memory bound and can benefit from larger values of S; however, the optimization of each model is not our main concern. Note this image by which PASCAL images are an average of 384 × 473 pixels and thus the single-scale setting typically upsamples by 1.6 times. The average effective stride at the return on investment (ROI) pooling layer is therefore ≈ 10 pixels.

In the multi-scale setting, we use the same five scales specified in [11] (s ∈ {480, 576, 688, 864, 1200}) to facilitate comparison with SPPnet. However, we cap the longest side at 2000 pixels to avoid exceeding GPU memory.

In the multi - scale setting, using the same five weighing scales specified in [11] (S ∈ {480, 576, 688, 864, 1200}) facilitates comparison with SPPnet. However, we hat the long side of 2000 pixels so as not to exceed GPU memory.

Table 7 shows models S and M when trained and tested with either one or five scales. Perhaps the most surprising result in [11] was that single-scale detection performs almost as well as multi-scale detection. Our findings con firm their result: deep ConvNets are adept at directly learning scale invariance. The multi-scale approach offers only a small increase in mAP at a large cost in compute time (Table 7). In the case of VGG16 (model L), we are limited to using a single scale by implementation details. Yet it achieves a mAP of 66.9%, which is slightly higher than the 66.0% reported for R-CNN [10], even though R-CNN uses “infinite” scales in the sense that each proposal is warped to a canonical size.

Table 7 shows models S and M when tested with one of the five trained weighing scales. Perhaps the most surprising result in [11] was that single-size detection performs as well as multi-scale detection. Our findings firm up the convicts' result: deep ConvNets are scale invariance learning directly from skilled. The multi-scale approach gives only a small increase to the map at a large cost in computation time (Table 7). In the case of VGG16 (model L), we are limited to using a single scale by a detailed implementation. Yet it can get a map of 66.9 %, slightly higher than the 66.0 % reported for R-CNN [10], even though R-CNN uses "infinite" weighing scales in the sense that each proposal is warped to a regular size.

Since single-scale processing offers the best tradeoff between speed and accuracy, especially for very deep models, all experiments outside of this sub-section use single-scale training and testing with s = 600 pixels.

Since single-scale processing provides the optimal balance between speed and accuracy, especially for very deep models, all experiments of this subsection, from outside, = use 600-pixel single-scale training and testing.

A good object detector should improve when supplied with more training data. Zhu et al. [24] found that DPM [8] mAP saturates after only a few hundred to thousand training examples. Here we augment the VOC07 trainval set with the VOC12 trainval set, roughly tripling the number of images to 16.5k, to evaluate Fast R-CNN. Enlarging the training set improves mAP on VOC07 test from 66.9% to 70.0% (Table 1). When training on this dataset we use 60k mini-batch iterations instead of 40k.

A good purpose detector is expected to improve when supplied with more training data. Scarlet et al. [24] found that the DPM [8] map saturates after a hundred to thousand few training examples. Here, to evaluate fast R-CNN, we increase the VOC12 trainval set set trainval VOC07, doubling the number of images to about 16.5k. Enlarging the training set improves the map on the VOC07 test from 66.9 % to 70.0 % (Table 1). When training on this dataset, we use 60K mini batch iterations instead of 40K.

We perform similar experiments for VOC10 and 2012, for which we construct a dataset of 21.5k images from the union of VOC07 trainval, test, and VOC12 trainval. When training on this dataset, we use 100k SGD iterations and lower the learning rate by 0.1× each 40k iterations (instead of each 30k). For VOC10 and 2012, mAP improves from 66.1% to 68.8% and from 65.7% to 68.4%, respectively.

We perform similar experiments for VOC10 and 2012, in which we construct a dataset of 21.5k images from the union of VOC07 trainval, test and VOC12 trainval. When training on this dataset, we use 100,000 SGD iterations and lower the learning rate by 0.1 × each 40K iterations (instead of each 30K). The VOC10 and 2012 map improves from 66.1 % to 68.8 % and from 65.7 % to 68.4 %, respectively.

Fast R-CNN uses the softmax classifier learnt during fine-tuning instead of training one-vs-rest linear SVMs post-hoc, as was done in R-CNN and SPPnet. To understand the impact of this choice, we implemented post-hoc SVM training with hard negative mining in Fast R-CNN. We use the same training algorithm and hyper-parameters as in R-CNN.

Fast R-CNN instead uses the softmax classifier learned during the fine-tuning of trained one-vs-pedestal, linear SVMs post-hoc, was done in R-CNN and SPPnet. To understand the impact of this choice, we implemented post-hoc SVM training with hard negative mines in fast R-CNN. We use the same training algorithm and hyper parameters as R-CNN.

Table 8 shows softmax slightly outperforming SVM for all three networks, by +0.1 to +0.8 mAP points. This effect is small, but it demonstrates that “one-shot” fine-tuning is sufficient compared to previous multi-stage training approaches. We note that softmax, unlike one-vs-rest SVMs, introduces competition between classes when scoring a RoI.

Table 8 shows softmax slightly outperforming SVM for three networks, 0.1-0.8 map points. This effect is small, but it shows that "one-shot" fine-tuning is sufficient compared to previous multi-stage training approaches. We note that when scoring a return on investment (ROI), softmax, unlike one-vs-pedestal SVMs, introduces competition between classes.

There are (broadly) two types of object detectors: those that use a sparse set of object proposals (e.g., selective search [21]) and those that use a dense set (e.g., DPM [8]). Classifying sparse proposals is a type of cascade [22] in which the proposal mechanism first rejects a vast number of candidates leaving the classifier with a small set to evaluate. This cascade improves detection accuracy when applied to DPM detections [21]. We find evidence that the proposalclassifier cascade also improves Fast R-CNN accuracy.

(Broad) there are two types of object detectors: those using a sparse set of object proposals (e.g., selective search [21]) and those using a dense set (e.g., DPM [8]). Classifying sparse proposals is a form of cascade [22] in which the mechanism first rejects numerous candidates, leaving the classification with a small set to evaluate. When applied to DPM detection [21], this waterfall improves detection accuracy. We can find evidence that the proposalclassifier waterfall improves fast R-CNN accuracy.

Using selective search’s quality mode, we sweep from 1k to 10k proposals per image, each time re-training and retesting model M. If proposals serve a purely computational role, increasing the number of proposals per image should not harm mAP

Using selective search's quality mode, we, cleaning from proposals of 1,000 to -10 thousand per image, each time re-training and retesting model M. If proposals perform a purely computational role, increasing the number of proposals per image should not harm the map

We find that mAP rises and then falls slightly as the proposal count increases (Fig. 3, solid blue line). This experiment shows that swamping the deep classifier with more proposals does not help, and even slightly hurts, accuracy.

We find the map rises and then falls slightly as the proposal count increases (Fig. 3, blue line). This experiment shows that burdening the deep classification with more proposals does not help, and accuracy is in pain even a little.

This result is difficult to predict without actually running the experiment. The state-of-the-art for measuring object proposal quality is Average Recall (AR) [12]. AR correlates well with mAP for several proposal methods using R-CNN, when using a fixed number of proposals per image. Fig. 3 shows that AR (solid red line) does not correlate well with mAP as the number of proposals per image is varied. AR must be used with care; higher AR due to more proposals does not imply that mAP will increase. Fortunately, training and testing with model M takes less than 2.5 hours. Fast R-CNN thus enables efficient, direct evaluation of object proposal mAP, which is preferable to proxy metrics.

This result is difficult to predict without running an actual experiment. The state of the art for measuring object proposal quality is average recall (AR) [12]. AR correlates well with the map for some proposal methods using R-CNN, when using a fixed number of proposals per image. Fig. 3 showed that AR (solid-state red line) has no correlation with the map as the number of proposals per image varies. AR must be used with care; higher AR due to more proposals does not mean the map increases. Fortunately, training and testing with model M takes less than 2.5 hours. Fast R-CNN thus allows efficient, direct evaluation of the object proposal map, for which proxy measurement is preferable.

We also investigate Fast R-CNN when using densely generated boxes (over scale, position, and aspect ratio), at a rate of about 45k boxes / image. This dense set is rich enough that when each selective search box is replaced by its closest (in IoU) dense box, mAP drops only 1 point (to 57.7%, Fig. 3, blue triangle). The statistics of the dense boxes differ from those of selective search boxes. Starting with 2k selective search boxes, we test mAP when adding a random sample of 1000 × {2, 4, 6, 8, 10, 32, 45} dense boxes. For each experiment we re-train and re-test model M. When these dense boxes are added, mAP falls more strongly than when adding more selective search boxes, eventually reaching 53.0%.

/ We also investigate fast R-CNN when using densely generated boxes (over scale, position, aspect ratio) at a rate of 45K boxes for image. This dense set is rich enough when each selective search box is replaced by the dense box closest (in IOU); the map deletes only 1 point (to 57.7 %, Fig. 3, blue triangle). The statistics of dense boxes differ from selective search boxes. Starting from 2K selective search boxes, the map is tested when adding a random sample of 1000 × {2, 4, 6, 8, 10, 32, 45} dense boxes. For each experiment, we again train adding these dense boxes, and when we re-test model M., the map falls more strongly than when adding more selective search windows, eventually reaching 53.0 %.

We also train and test Fast R-CNN using only dense boxes (45k / image). This setting yields a mAP of 52.9% (blue diamond). Finally, we check if SVMs with hard negative mining are needed to cope with the dense box distribution. SVMs do even worse: 49.3% (blue circle).

We also test fast R-CNN using training only dense boxes (45K / image). This setting yields a map of 52.9 % (blue diamond). If SVMs with hard negative mines are needed to cope with the dense box distribution, finally, we check. SVMs perform worsening 49.3 % (blue circle).

We applied Fast R-CNN (with VGG16) to the MS COCO dataset [18] to establish a preliminary baseline. We trained on the 80k image training set for 240k iterations and evaluated on the “test-dev” set using the evaluation server. The PASCAL-style mAP is 35.9%; the new COCO-style AP, which also averages over IoU thresholds, is 19.7%.

We applied fast R-CNN (with VGG16) to the MS Coco dataset [18] to set a preliminary baseline. We train on the 80K image training set of 240K iterations and set "test dev in" evaluation using the evaluation server. The Pascal-style map is 35.9 %

This paper proposes Fast R-CNN, a clean and fast update to R-CNN and SPPnet. In addition to reporting state-of-theart detection results, we present detailed experiments that we hope provide new insights. Of particular note, sparse object proposals appear to improve detector quality. This issue was too costly (in time) to probe in the past, but becomes practical with Fast R-CNN. Of course, there may exist yet undiscovered techniques that allow dense boxes to perform as well as sparse proposals. Such methods, if developed, may help further accelerate object detection.

This paper proposes fast R-CNN, a clean and fast update to R-CNN and SPPnet. In addition to reporting the country's theart detection results, we present detailed experiments that we hope provide new insights. In particular, for reference, sparse object proposals appear to improve detector quality. This problem was too costly (in time) to probe in the past, but becomes reality with fast R-CNN. Of course, not-yet-discovered techniques may exist that let dense boxes perform as well as sparse proposals. Developing such methods can further help accelerate object detection.

Acknowledgements. I thank Kaiming He, Larry Zitnick, and Piotr Dollar for helpful discussions and encouragement.

Acknowledgements. I thank Kaiming he, Larry Zitnick, and Pyotr dollar for helpful discussion and encouragement.