MaskRCNN:Paper

We present a conceptually simple, flexible, and general framework for object instance segmentation. Our approach efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. Mask R-CNN is simple to train and adds only a small overhead to Faster R-CNN, running at 5 fps. Moreover, Mask R-CNN is easy to generalize to other tasks, e.g., allowing us to estimate human poses in the same framework. We show top results in all three tracks of the COCO suite of challenges, including instance segmentation, boundingbox object detection, and person keypoint detection. Without bells and whistles, Mask R-CNN outperforms all existing, single-model entries on every task, including the COCO 2016 challenge winners. We hope our simple and effective approach will serve as a solid baseline and help ease future research in instance-level recognition. Code has been made available at: https://github.com/facebookresearch/Detectron

We present a conceptually simple, flexible, and general framework for object instance segmentation. Our approach detects objects in an image efficiently while also generating a high-quality segmentation mask for each instance. This method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting object masks, in parallel with the existing branch for bounding box recognition. Mask R-CNN is easy to train and adds only a small overhead to Faster R-CNN, which runs at 5 fps. Mask R-CNN is also easy to generalize to other tasks. For example, it can estimate human poses within the same framework. It shows top results in all three tracks of the COCO challenge suite, including instance segmentation, bounding box object detection, and individual keypoint detection. Without the bells and whistles, Mask R-CNN outperforms every existing single-model entry on all tasks, including the COCO 2016 challenge winners. We hope this simple and effective approach becomes a solid baseline and makes future research on instance-level recognition easier. The code is available at https://github.com/facebookresearch/Detectron.

The vision community has rapidly improved object detection and semantic segmentation results over a short period of time. In large part, these advances have been driven by powerful baseline systems, such as the Fast/Faster RCNN⁴⁶⁴⁷ and Fully Convolutional Network (FCN) ⁴⁸ frameworks for object detection and semantic segmentation, respectively. These methods are conceptually intuitive and offer flexibility and robustness, together with fast training and inference time. Our goal in this work is to develop a comparably enabling framework for instance segmentation.

The vision community has quickly improved object detection and semantic segmentation results over a short time. By and large, these advances came from powerful baseline systems such as the Fast / Faster RCNN⁴⁹⁵⁰ and FNN (Fully Convolutional Network) ⁵¹ frameworks, for object detection and semantic segmentation respectively. These methods are conceptually intuitive and provide flexibility and robustness along with fast training and inference time. The goal of this work is to develop a comparable framework for, for example, segmentation.

Instance segmentation is challenging because it requires the correct detection of all objects in an image while also precisely segmenting each instance. It therefore combines elements from the classical computer vision tasks of object detection, where the goal is to classify individual objects and localize each using a bounding box, and semantic segmentation, where the goal is to classify each pixel into a fixed set of categories without differentiating object instances.1 Given this, one might expect a complex method is required to achieve good results. However, we show that a surprisingly simple, flexible, and fast system can surpass prior state-of-the-art instance segmentation results.

Instance segmentation is tricky because it must detect every object in an image accurately while also segmenting each instance accurately. It therefore combines elements of the classical computer vision task of object detection. Here the goal is to classify individual objects and localize each one using a bounding box. Semantic classification classifies each pixel into a fixed set of categories without distinguishing them. Given this, one might expect that a complex method is needed to get good results. However, we show that a surprisingly simple, flexible, and fast system can surpass existing state-of-the-art instance segmentation results.

Our method, called Mask R-CNN, extends Faster R-CNN ⁵² by adding a branch for predicting segmentation masks on each Region of Interest (RoI), in parallel with the existing branch for classification and bounding box regression (Figure 1). The mask branch is a small FCN applied to each RoI, predicting a segmentation mask in a pixel-topixel manner. Mask R-CNN is simple to implement and train given the Faster R-CNN framework, which facilitates a wide range of flexible architecture designs. Additionally, the mask branch only adds a small computational overhead, enabling a fast system and rapid experimentation.

This method, called Mask R-CNN, extends Faster R-CNN by adding a branch for predicting a segmentation mask on each region of interest (RoI), alongside the existing branch for classification and bounding box regression ⁵³ (Figure 1). . The mask branch is a small FCN applied to each RoI, and it predicts the segmentation mask in a pixel-to-pixel manner. Mask R-CNN is simple to implement and train through the more flexible R-CNN framework, so a wide range of flexible architecture designs is possible. In addition, the mask branch adds only a small computational overhead, which makes a fast system and rapid experimentation possible.

In principle Mask R-CNN is an intuitive extension of Faster R-CNN, yet constructing the mask branch properly is critical for good results. Most importantly, Faster RCNN was not designed for pixel-to-pixel alignment between network inputs and outputs. This is most evident in how RoIPool⁵⁴⁵⁵, the de facto core operation for attending to instances, performs coarse spatial quantization for feature extraction. To fix the misalignment, we propose a simple, quantization-free layer, called RoIAlign, that faithfully preserves exact spatial locations. Despite being a seemingly minor change, RoIAlign has a large impact: it improves mask accuracy by relative 10% to 50%, showing bigger gains under stricter localization metrics. Second, we found it essential to decouple mask and class prediction: we predict a binary mask for each class independently, without competition among classes, and rely on the network’s RoI classification branch to predict the category. In contrast, FCNs usually perform per-pixel multi-class categorization, which couples segmentation and classification, and based on our experiments works poorly for instance segmentation.

In principle, Mask R-CNN is an intuitive extension of Faster R-CNN, but building the mask branch correctly matters for good results. Most importantly, Faster RCNN was not designed for pixel-to-pixel alignment between network inputs and outputs. This is most obvious in how RoIPool⁵⁶⁵⁷, the de facto core operation for attending to instances, performs coarse spatial quantization for feature extraction. To fix the misalignment, we propose a simple, quantization-free layer called RoIAlign that faithfully preserves exact spatial locations. Despite being a seemingly minor change, RoIAlign has a big impact. It improves mask accuracy from 10% to 50%, showing larger gains under strict localization metrics. Second, we found that separating mask and class prediction is essential. We predict a binary mask for each class independently, without competition between classes, and use the network's RoI classification branch to predict the category. In contrast, FCNs usually perform per-pixel multi-class classification, and this classification combines classification and classification; according to our experiments it is not suited to segmentation.

Without bells and whistles, Mask R-CNN surpasses all previous state-of-the-art single-model results on the COCO instance segmentation task ⁵⁸, including the heavilyengineered entries from the 2016 competition winner. As a by-product, our method also excels on the COCO object detection task. In ablation experiments, we evaluate multiple basic instantiations, which allows us to demonstrate its robustness and analyze the effects of core factors.

Without the bells and whistles, Mask R-CNN surpasses every latest state-of-the-art single-model result on the COCO instance segmentation task ⁵⁹, including the engineers who won the 2016 competition. As a by-product, our method also excels at the COCO object detection task. In the ablation experiments, we evaluate several basic instantiations, which lets us prove robustness and analyze the effects of the core factors.

Our models can run at about 200ms per frame on a GPU, and training on COCO takes one to two days on a single 8-GPU machine. We believe the fast train and test speeds, together with the framework’s flexibility and accuracy, will benefit and ease future research on instance segmentation.

Our model can run at about 200ms per frame on a GPU, and training on COCO takes 1-2 days on a single 8-GPU system. We believe the fast train and test speeds, together with the flexibility and accuracy of the framework, will help future research on instance segmentation.

Finally, we showcase the generality of our framework via the task of human pose estimation on the COCO keypoint dataset ⁶⁰. By viewing each keypoint as a one-hot binary mask, with minimal modification Mask R-CNN can be applied to detect instance-specific poses. Mask R-CNN surpasses the winner of the 2016 COCO keypoint competition, and at the same time runs at 5 fps. Mask R-CNN, therefore, can be seen more broadly as a flexible framework for instance-level recognition and can be readily extended to more complex tasks. We have released code to facilitate future research.

Finally, we show the generality of the framework through the task of human pose estimation on the COCO keypoint dataset ⁶¹. By viewing each keypoint as a one-hot binary mask, Mask R-CNN can be applied with minimal modification to detect instance-specific poses. Mask R-CNN surpasses the winner of the 2016 COCO keypoint competition and runs at 5fps at the same time. Mask R-CNN can therefore be seen more broadly as a flexible framework for instance-level recognition, and it can easily be extended to more complex tasks. We have released code that makes future research easier.

R-CNN: The Region-based CNN (R-CNN) approach ⁶² to bounding-box object detection is to attend to a manageable number of candidate object regions⁶³⁶⁴ and evaluate convolutional networks⁶⁵⁶⁶ independently on each RoI. R-CNN was extended⁶⁷⁶⁸ to allow attending to RoIs on feature maps using RoIPool, leading to fast speed and better accuracy. Faster R-CNN ⁶⁹ advanced this stream by learning the attention mechanism with a Region Proposal Network (RPN). Faster R-CNN is flexible and robust to many follow-up improvements (e.g.,⁷⁰⁷¹⁷²), and is the current leading framework in several benchmarks.

R-CNN: The region-based CNN (R-CNN) approach ⁷³ to bounding box object detection is to attend to a manageable number of candidate object regions⁷⁴⁷⁵ and evaluate them. Convolutional networks⁷⁶⁷⁷ independently on each RoI. R-CNN was extended so that it can attend to RoIs on feature maps using RoIPool⁷⁸⁷⁹, improving speed and accuracy. Fast R-CNN ⁸⁰ advanced this stream by learning an attention mechanism through a region proposal network (RPN). Faster R-CNN is flexible and robust to many follow-up improvements (e.g.:⁸¹⁸²⁸³), and it is currently the leading framework in several benchmarks.

Instance Segmentation: Driven by the effectiveness of RCNN, many approaches to instance segmentation are based on segment proposals. Earlier methods^84858687 resorted to bottom-up segments⁸⁸⁸⁹. DeepMask ⁹⁰ and following works⁹¹⁹² learn to propose segment candidates, which are then classified by Fast R-CNN. In these methods, segmentation precedes recognition, which is slow and less accurate. Likewise, Dai et al. ⁹³ proposed a complex multiple-stage cascade that predicts segment proposals from bounding-box proposals, followed by classification. Instead, our method is based on parallel prediction of masks and class labels, which is simpler and more flexible.

Instance segmentation: Following the effectiveness of RCNN, many approaches to instance segmentation are based on segment proposals. Earlier methods^94959697 relied on bottom-up segments⁹⁸⁹⁹. DeepMask ¹⁰⁰ and the works that followed¹⁰¹¹⁰² learn how to propose segment candidates, which are classified with Fast R-CNN. In these methods, segmentation comes before recognition, and it is slow and less accurate. Likewise, Dai et al. ¹⁰³ proposed a complex multi-stage cascade that predicts segment proposals from bounding box proposals and then performs classification. Instead, our method is based on parallel prediction of masks and class labels, which is simpler and more flexible.

Most recently, Li et al. ¹⁰⁴ combined the segment proposal system in ¹⁰⁵ and object detection system in ¹⁰⁶ for “fully convolutional instance segmentation” (FCIS). The common idea in^107108109 is to predict a set of positionsensitive output channels fully convolutionally. These channels simultaneously address object classes, boxes, and masks, making the system fast. But FCIS exhibits systematic errors on overlapping instances and creates spurious edges (Figure 6), showing that it is challenged by the fundamental difficulties of segmenting instances.

Most recently, Li et al. ¹¹⁰ combined the segment proposal system of ¹¹¹ and the object detection system of ¹¹² for "fully convolutional instance segmentation" (FCIS).The common idea of ^113114115 is to predict a set of position-sensing output channels in a fully convolutional way. These channels handle object classes, boxes, and masks at the same time, which makes the system fast. However, FCIS produces systematic errors on overlapping instances and creates spurious edges (Figure 6), showing that it struggles because of the fundamental difficulty of instance segments.

Another family of solutions^116117118119 to instance segmentation are driven by the success of semantic segmentation. Starting from per-pixel classification results (e.g., FCN outputs), these methods attempt to cut the pixels of the same category into different instances. In contrast to the segmentation-first strategy of these methods, Mask R-CNN is based on an instance-first strategy. We expect a deeper incorporation of both strategies will be studied in the future.

Another family of solutions to instance segmentation (23, 4, 3, 29) is driven by the success of semantic segmentation. Starting from per-pixel classification results (e.g.: FCN outputs), these methods cut the pixels of the same category into different instances. Unlike the segmentation-first strategy of these methods, Mask R-CNN is based on an instance-first strategy. We expect the integration of the two strategies to be studied more deeply in the future.

Mask R-CNN is conceptually simple: Faster R-CNN has two outputs for each candidate object, a class label and a bounding-box offset; to this we add a third branch that outputs the object mask. Mask R-CNN is thus a natural and intuitive idea. But the additional mask output is distinct from the class and box outputs, requiring extraction of much finer spatial layout of an object. Next, we introduce the key elements of Mask R-CNN, including pixel-to-pixel alignment, which is the main missing piece of Fast/Faster R-CNN.

Mask R-CNN is conceptually simple. Fast R-CNN has two outputs for each candidate object, a class label and a bounding box offset. We add a third branch to this that outputs the object mask. Mask R-CNN is therefore a natural and intuitive idea. However, the additional mask output differs from the class and box outputs, so it requires extracting a much finer spatial layout of the object. Next, we introduce the key elements of Mask R-CNN, including pixel-to-pixel alignment, the main missing piece of Fast / Faster R-CNN.

Faster R-CNN: We begin by briefly reviewing the Faster R-CNN detector ¹²⁰. Faster R-CNN consists of two stages. The first stage, called a Region Proposal Network (RPN), proposes candidate object bounding boxes. The second stage, which is in essence Fast R-CNN ¹²¹, extracts features using RoIPool from each candidate box and performs classification and bounding-box regression. The features used by both stages can be shared for faster inference. We refer readers to ¹²² for latest, comprehensive comparisons between Faster R-CNN and other frameworks.

Faster R-CNN: We start with a brief review of the Faster R-CNN detector ¹²³. Fast R-CNN consists of two stages. The first stage, called the region proposal network (RPN), proposes candidate object bounding boxes. The second stage, which is essentially Fast R-CNN ¹²⁴, extracts features from each candidate box using RoIPool and performs classification and bounding box regression. The features used in both stages can be shared for faster inference. For the latest comprehensive comparison between Faster R-CNN and other frameworks, readers should refer to ¹²⁵.

Mask R-CNN: Mask R-CNN adopts the same two-stage procedure, with an identical first stage (which is RPN). In the second stage, in parallel to predicting the class and box offset, Mask R-CNN also outputs a binary mask for each RoI. This is in contrast to most recent systems, where classification depends on mask predictions (e.g.^126127128). Our approach follows the spirit of Fast R-CNN ¹²⁹ that applies bounding-box classification and regression in parallel (which turned out to largely simplify the multi-stage pipeline of original R-CNN ¹³⁰).

Mask R-CNN: Mask R-CNN adopts the same two-stage procedure, with the same first stage (RPN). In the second stage, in parallel with the class and box offset prediction, Mask R-CNN outputs a binary mask for each RoI. This contrasts with most recent systems, where classification depends on mask prediction (e.g.:^131132133). Our approach follows the spirit of Fast R-CNN ¹³⁴, which applies bounding box classification and regression in parallel (this turned out to greatly simplify the multi-stage pipeline of the original R-CNN ¹³⁵).

Formally, during training, we define a multi-task loss on each sampled RoI as L = Lcls + Lbox + Lmask. The classification loss Lcls and bounding-box loss Lbox are identical as those defined in ¹³⁶. The mask branch has a Km2 - dimensional output for each RoI, which encodes K binary masks of resolution m × m, one for each of the K classes. To this we apply a per-pixel sigmoid, and define Lmask as the average binary cross-entropy loss. For an RoI associated with ground-truth class k, Lmask is only defined on the k-th mask (other mask outputs do not contribute to the loss).

Formally, during training we define the multi-task loss for each sampled RoI as L = Lcls + Lbox + Lmask. The classification loss Lcl and the bounding box loss Lbox are the same as those defined in ¹³⁷. The mask branch has a Km2-dimensional output for each RoI, encoding K binary masks of resolution m × m, one for each of the K classes. For this we apply a per-pixel sigmoid and define Lmask as the average binary cross-entropy loss. For an RoI associated with ground-truth class k, Lmask is defined only on the k-th mask (the other mask outputs do not contribute to the loss).

Our definition of Lmask allows the network to generate masks for every class without competition among classes; we rely on the dedicated classification branch to predict the class label used to select the output mask. This decouples mask and class prediction. This is different from common practice when applying FCNs ¹³⁸ to semantic segmentation, which typically uses a per-pixel softmax and a multinomial cross-entropy loss. In that case, masks across classes compete; in our case, with a per-pixel sigmoid and a binary loss, they do not. We show by experiments that this formulation is key for good instance segmentation results.

The definition of Lmask lets the network generate masks for every class without competition between classes. A dedicated classification branch is used to predict the class label used to select the output mask. Mask and class prediction are separated. This differs from the usual practice when applying FCN ¹³⁹ to semantic segmentation, which generally uses a per-pixel softmax and a multinomial cross-entropy loss. In that case, masks across classes compete. In our case, with a per-pixel sigmoid and a binary loss, they do not. We show through experiments that this formulation is the key to good instance segmentation results.

Mask Representation: A mask encodes an input object’s spatial layout. Thus, unlike class labels or box offsets that are inevitably collapsed into short output vectors by fully-connected (fc) layers, extracting the spatial structure of masks can be addressed naturally by the pixel-to-pixel correspondence provided by convolutions.

Mask representation: A mask encodes the spatial layout of the input object. So, unlike class labels or box offsets, which fully connected (fc) layers inevitably reduce to short output vectors, extracting the spatial structure of a mask can be handled naturally by the pixel-to-pixel correspondence that convolution provides.

Specifically, we predict an m × m mask from each RoI using an FCN ¹⁴⁰. This allows each layer in the mask branch to maintain the explicit m × m object spatial layout without collapsing it into a vector representation that lacks spatial dimensions. Unlike previous methods that resort to fc layers for mask prediction^141142143, our fully convolutional representation requires fewer parameters, and is more accurate as demonstrated by experiments.

In particular, we use an FCN to predict an m × m mask from each RoI ¹⁴⁴. This lets each layer of the mask branch keep an explicit m × m object spatial layout without reducing it to a vector representation that has no spatial dimension. Unlike earlier methods that rely on fc layers for mask prediction^145146147, our fully convolutional representation needs fewer parameters and is more accurate according to experiments.

This pixel-to-pixel behavior requires our RoI features, which themselves are small feature maps, to be well aligned to faithfully preserve the explicit per-pixel spatial correspondence. This motivated us to develop the following RoIAlign layer that plays a key role in mask prediction.

This pixel-to-pixel behavior requires the RoI features, which are themselves small feature maps, to be well aligned in order to faithfully preserve the per-pixel spatial correspondence. This led us to develop the following RoIAlign layer, which plays a key role in mask prediction.

RoIAlign: RoIPool ¹⁴⁸ is a standard operation for extracting a small feature map (e.g., 7×7) from each RoI. RoIPool first quantizes a floating-number RoI to the discrete granularity of the feature map, this quantized RoI is then subdivided into spatial bins which are themselves quantized, and finally feature values covered by each bin are aggregated (usually by max pooling). Quantization is performed, e.g., on a continuous coordinate x by computing [x/16], where 16 is a feature map stride and [·] is rounding; likewise, quantization is performed when dividing into bins (e.g., 7×7). These quantizations introduce misalignments between the RoI and the extracted features. While this may not impact classification, which is robust to small translations, it has a large negative effect on predicting pixel-accurate masks.

RoIAlign: RoIPool ¹⁴⁹ is the standard operation for extracting a small feature map (e.g.: 7x7) from each RoI. RoIPool first quantizes the floating-number RoI to the discrete units of the feature map, then subdivides this quantized RoI into spatial bins, quantizing them as well, and then aggregates the feature values that each bin covers (usually max pooling). Quantization is performed on a continuous coordinate x, for example by computing [x / 16], where 16 is the feature map stride and [·] rounds; likewise, quantization is performed when dividing into bins (e.g., 7 x 7). These quantizations cause misalignment between the RoI and the extracted features. This does not affect classification, which is robust to small translations, but it has a large negative effect on predicting pixel-accurate masks.

To address this, we propose an RoIAlign layer that removes the harsh quantization of RoIPool, properly aligning the extracted features with the input. Our proposed change is simple: we avoid any quantization of the RoI boundaries or bins (i.e., we use x/16 instead of [x/16]). We use bilinear interpolation ¹⁵⁰ to compute the exact values of the input features at four regularly sampled locations in each RoI bin, and aggregate the result (using max or average), see Figure 3 for details. We note that the results are not sensitive to the exact sampling locations, or how many points are sampled, as long as no quantization is performed.

To solve this, we propose an RoIAlign layer that removes the harsh quantization of RoIPool and correctly aligns the extracted features with the input. The proposed change is simple. Avoid quantizing the RoI boundaries or bins (that is, we use x / 16 instead of [x / 16]). We use bilinear interpolation ¹⁵¹ to compute the exact values of the input features at 4 regularly sampled locations in each RoI bin and aggregate the results (using max or average). See Figure 3 for details. As long as no quantization is performed, the results are not sensitive to the exact sampling locations or the number of sampled points.

RoIAlign leads to large improvements as we show in §4.2. We also compare to the RoIWarp operation proposed in ¹⁵². Unlike RoIAlign, RoIWarp overlooked the alignment issue and was implemented in ¹⁵³ as quantizing RoI just like RoIPool. So even though RoIWarp also adopts bilinear resampling motivated by ¹⁵⁴, it performs on par with RoIPool as shown by experiments (more details in Table 2c), demonstrating the crucial role of alignment.

RoIAlign has improved greatly, as shown in §4.2. We also compare it with the RoIWarp operation proposed in ¹⁵⁵. Unlike RoIAlign, RoIWarp overlooks the alignment problem and was implemented in ¹⁵⁶ in a way that quantizes the RoI just like RoIPool. So even though RoIWarp adopts bilinear resampling motivated by ¹⁵⁷, it performs on par with RoIPool as the experiments show (see the details in Table 2c), which shows the important role of alignment.

Network Architecture: To demonstrate the generality of our approach, we instantiate Mask R-CNN with multiple architectures. For clarity, we differentiate between: (i) the convolutional backbone architecture used for feature extraction over an entire image, and (ii) the network head for bounding-box recognition (classification and regression) and mask prediction that is applied separately to each RoI.

Network architecture: To prove the generality of the approach, we instantiate Mask R-CNN with several architectures. For clarity, we distinguish as follows: (i) the convolutional backbone architecture used for feature extraction over the whole image, and (ii) the network head for bounding box recognition (classification and regression) and the mask prediction applied separately to each RoI.

We denote the backbone architecture using the nomenclature network-depth-features. We evaluate ResNet ¹⁵⁸ and ResNeXt ¹⁵⁹ networks of depth 50 or 101 layers. The original implementation of Faster R-CNN with ResNets ¹⁶⁰ extracted features from the final convolutional layer of the 4-th stage, which we call C4. This backbone with ResNet-50, for example, is denoted by ResNet-50-C4. This is a common choice used in^161162163164.

We denote the backbone architecture using the nomenclature network-depth-features. We evaluate ResNet ¹⁶⁵ and ResNeXt ¹⁶⁶ networks with depths of 50 or 101 layers. The original implementation of Faster R-CNN using ResNets extracted features from the last convolutional layer of stage 4, which ¹⁶⁷ calls C4. For example, this backbone with ResNet-50 is denoted ResNet-50-C4. This is a choice commonly used in^168169170171.

We also explore another more effective backbone recently proposed by Lin et al. ¹⁷², called a Feature Pyramid Network (FPN). FPN uses a top-down architecture with lateral connections to build an in-network feature pyramid from a single-scale input. Faster R-CNN with an FPN backbone extracts RoI features from different levels of the feature pyramid according to their scale, but otherwise the rest of the approach is similar to vanilla ResNet. Using a ResNet-FPN backbone for feature extraction with Mask RCNN gives excellent gains in both accuracy and speed. For further details on FPN, we refer readers to ¹⁷³.

We also Lin et al. ¹⁷⁴, called the feature pyramid network (FPN). FPN uses a top-down architecture with lateral connections to build an in-network feature pyramid from a single-scale input. Faster R-CNN with an FPN backbone extracts RoI features from various levels of the feature pyramid according to scale, but otherwise the rest of the approach is similar to vanilla ResNet. Using a ResNet-FPN backbone for feature extraction with Mask RCNN greatly improves both accuracy and speed. For details on FPN, readers should refer to ¹⁷⁵.

For the network head we closely follow architectures presented in previous work to which we add a fully convolutional mask prediction branch. Specifically, we extend the Faster R-CNN box heads from the ResNet ¹⁷⁶ and FPN ¹⁷⁷ papers. Details are shown in Figure 4. The head on the ResNet-C4 backbone includes the 5-th stage of ResNet (namely, the 9-layer ‘res5’ ¹⁷⁸), which is computeintensive. For FPN, the backbone already includes res5 and thus allows for a more efficient head that uses fewer filters.

For the network head, we strictly follow the architectures presented in previous work and add a fully convolutional mask prediction branch. Specifically, we extend the Faster R-CNN box heads from the ResNet ¹⁷⁹ and FPN ¹⁸⁰ papers. Details are shown in Figure 4. The head of the ResNet-C4 backbone includes stage 5 of ResNet (that is, the 9-layer 'res5'¹⁸¹), which is compute-intensive. For FPN, the backbone already includes res5, so a more efficient head that uses fewer filters is possible.

We note that our mask branches have a straightforward structure. More complex designs have the potential to improve performance but are not the focus of this work.

The mask branch has a simple structure. More complex designs have the potential to improve performance, but they are not the focus of this work.

We set hyper-parameters following existing Fast/Faster R-CNN work^182183184. Although these decisions were made for object detection in original papers^185186187, we found our instance segmentation system is robust to them.

We set the hyperparameters following existing Fast / Faster R-CNN work^188189190. These decisions were made for object detection in the original papers^191192193, but we found that the instance segmentation system is robust.

Training: As in Fast R-CNN, an RoI is considered positive if it has IoU with a ground-truth box of at least 0.5 and negative otherwise. The mask loss Lmask is defined only on positive RoIs. The mask target is the intersection between an RoI and its associated ground-truth mask.

Training: As in Fast R-CNN, an RoI is considered positive if it is a ground-truth box with IoU of 0.5 or more, and negative otherwise. The mask loss Lmask is defined only on positive RoIs. The mask target is the intersection of the RoI and its associated ground-truth mask.

We adopt image-centric training ¹⁹⁴. Images are resized such that their scale (shorter edge) is 800 pixels ¹⁹⁵. Each mini-batch has 2 images per GPU and each image has N sampled RoIs, with a ratio of 1:3 of positive to negatives ¹⁹⁶. N is 64 for the C4 backbone (as in¹⁹⁷¹⁹⁸) and 512 for FPN (as in ¹⁹⁹). We train on 8 GPUs (so effective minibatch size is 16) for 160k iterations, with a learning rate of 0.02 which is decreased by 10 at the 120k iteration. We use a weight decay of 0.0001 and momentum of 0.9. With ResNeXt ²⁰⁰, we train with 1 image per GPU and the same number of iterations, with a starting learning rate of 0.01.

We adopt image-centric training ²⁰¹. Images are resized so that their size (the shorter edge) is 800 pixels ²⁰². Each mini-batch has 2 images per GPU, and each image has N: sampled RoIs, with a positive-to-negative ratio of 1 : 3 ²⁰³. N is 64 for the C4 backbone, as in ²⁰⁴²⁰⁵) and 512 for FPN (same as ²⁰⁶). We train 8 GPUs (effective mini-batch size 16) for 160k iterations, and the learning rate is 0.02, decreasing by 10 at the 120k iteration. We use a weight decay of 0.0001 and momentum of 0.9. With ResNeXt ²⁰⁷, training uses 1 image per GPU and the same number of iterations, with a learning rate of 0.01.

The RPN anchors span 5 scales and 3 aspect ratios, following ²⁰⁸. For convenient ablation, RPN is trained separately and does not share features with Mask R-CNN, unless specified. For every entry in this paper, RPN and Mask R-CNN have the same backbones and so they are shareable.

Following ²⁰⁹, the RPN anchors span 5 scales and 3 aspect ratios. For convenient ablation, RPN is trained separately and does not share features with Mask R-CNN unless specified. For every entry in this paper, RPN and Mask R-CNN have the same backbone, so they can be shared.

Inference: At test time, the proposal number is 300 for the C4 backbone (as in ²¹⁰) and 1000 for FPN (as in ²¹¹). We run the box prediction branch on these proposals, followed by non-maximum suppression ²¹². The mask branch is then applied to the highest scoring 100 detection boxes. Although this differs from the parallel computation used in training, it speeds up inference and improves accuracy (due to the use of fewer, more accurate RoIs). The mask branch can predict K masks per RoI, but we only use the k-th mask, where k is the predicted class by the classification branch. The m×m floating-number mask output is then resized to the RoI size, and binarized at a threshold of 0.5.

Inference: At test time, the proposal number is proposed as the C4 backbone (300) (from ²¹³) and FPN (1000) (from ²¹⁴). We run the box prediction branch on these proposals, and then there is no maximum suppression ²¹⁵. The mask branch is applied to the 100 highest-scoring detection boxes. This differs from the parallel computation used in training, but it speeds up inference and improves accuracy (because fewer, more accurate RoIs are used). The mask branch can predict K masks per RoI, but uses only the k-th mask. Here k is the class predicted by the classification branch. After that, the m × m floating-number mask output is resized to the RoI size and binarized at a threshold of 0.5.

Note that since we only compute masks on the top 100 detection boxes, Mask R-CNN adds a small overhead to its Faster R-CNN counterpart (e.g., ∼20% on typical models).

Because masks are computed only on the top 100 detection boxes, Mask R-CNN adds a slight overhead to its Faster R-CNN counterpart (e.g.: ~ 20% for typical models).

We perform a thorough comparison of Mask R-CNN to the state of the art along with comprehensive ablations on the COCO dataset ²¹⁶. We report the standard COCO metrics including AP (averaged over IoU thresholds), AP50, AP75, and APS, APM, APL (AP at different scales). Unless noted, AP is evaluating using mask IoU. As in previous work²¹⁷²¹⁸, we train using the union of 80k train images and a 35k subset of val images (trainval35k), and report ablations on the remaining 5k val images (minival). We also report results on test-dev ²¹⁹.

We thoroughly compare Mask R-CNN with the state of the art, along with comprehensive removals on the COCO dataset ²²⁰. We report standard COCO metrics including AP (averaged over IoU thresholds), AP50, AP75, and APS, APM, APL (AP at various scales). Unless noted otherwise, AP is evaluated using mask IoU. As in previous studies²²¹²²², we train using the union of 80k train images and a 35k subset of images (trainval35k), and report ablations on the remaining 5kval images (minimum). We also report results on test-dev ²²³.

We compare Mask R-CNN to the state-of-the-art methods in instance segmentation in Table 1. All instantiations of our model outperform baseline variants of previous state-of-the-art models. This includes MNC ²²⁴ and FCIS ²²⁵, the winners of the COCO 2015 and 2016 segmentation challenges, respectively. Without bells and whistles, Mask R-CNN with ResNet-101-FPN backbone outperforms FCIS+++ ²²⁶, which includes multi-scale train/test, horizontal flip test, and online hard example mining (OHEM) ²²⁷. While outside the scope of this work, we expect many such improvements to be applicable to ours.

We compare Mask R-CNN with the latest methods in instance segmentation in Table 1. Every instantiation of the model outperforms the baseline variants of earlier state-of-the-art models. These include MNC ²²⁸ and FCIS ²²⁹, the winners of the COCO 2015 and 2016 segmentation challenges. Without the bells and whistles, Mask R-CNN with a ResNet-101-FPN backbone outperforms FCIS +++ ²³⁰. This includes multi-scale train / test, horizontal flip test, and online hard example mining (OHEM) ²³¹. While this is outside the scope of this study, we expect such improvements to apply to us.

Mask R-CNN outputs are visualized in Figures 2 and 5. Mask R-CNN achieves good results even under challenging conditions. In Figure 6 we compare our Mask R-CNN baseline and FCIS+++ ²³². FCIS+++ exhibits systematic artifacts on overlapping instances, suggesting that it is challenged by the fundamental difficulty of instance segmentation. Mask R-CNN shows no such artifacts.

Mask R-CNN outputs are shown in Figures 2 and 5. Mask R-CNN gets excellent results even under tricky conditions. Figure 6 compares the Mask R-CNN baseline and FCIS +++ ²³³. FCIS +++ shows systematic artifacts on overlapping instances, so it struggles because of the fundamental difficulty of instance segmentation. Mask R-CNN does not show such artifacts.

We run a number of ablations to analyze Mask R-CNN. Results are shown in Table 2 and discussed in detail next.

We run a number of ablations to analyze Mask R-CNN. The results are shown in Table 2 and discussed in detail next.

Architecture: Table 2a shows Mask R-CNN with various backbones. It benefits from deeper networks (50 vs. 101) and advanced designs including FPN and ResNeXt. We note that not all frameworks automatically benefit from deeper or advanced networks (see benchmarking in ²³⁴).

Architecture: Table 2a shows Mask R-CNN with various backbones. It can take advantage of deeper networks (50 vs 101) and advanced designs including FPN and ResNeXt. Not every framework automatically benefits from deeper or more advanced networks (see the benchmarking in ²³⁵).

Multinomial vs. Independent Masks: Mask R-CNN decouples mask and class prediction: as the existing box branch predicts the class label, we generate a mask for each class without competition among classes (by a per-pixel sigmoid and a binary loss). In Table 2b, we compare this to using a per-pixel softmax and a multinomial loss (as commonly used in FCN ²³⁶). This alternative couples the tasks of mask and class prediction, and results in a severe loss in mask AP (5.5 points). This suggests that once the instance has been classified as a whole (by the box branch), it is sufficient to predict a binary mask without concern for the categories, which makes the model easier to train.

Multinomial vs. independent masks: Mask R-CNN separates mask and class prediction. When the existing box branch predicts the class label, a mask is generated for each class without competition over each pixel (per-pixel sigmoid and binary loss). In Table 2b, we compare this with using a per-pixel softmax and a multinomial loss (commonly used in FCN ²³⁷). This alternative combines the tasks of mask and class prediction, causing a severe loss in mask AP (5.5 points). This suggests that once an instance has been classified as a whole (by the box branch), predicting a binary mask without worrying about categories is enough, which makes the model easy to train.

Class-Specific vs. Class-Agnostic Masks: Our default instantiation predicts class-specific masks, i.e., one m×m mask per class. Interestingly, Mask R-CNN with classagnostic masks (i.e., predicting a single m×m output regardless of class) is nearly as effective: it has 29.7 mask AP vs. 30.3 for the class-specific counterpart on ResNet-50-C4. This further highlights the division of labor in our approach which largely decouples classification and segmentation.

Class-Specific vs. Class-Agnostic Masks: The default instantiation of the class predicts class-specific masks, that is, one m × m mask per class. Interestingly, Mask R-CNN with class-agnostic masks (that is, predicting a single mxm output regardless of class) is nearly effective. It has 29.7 mask AP vs 30.3 for the class-specific counterpart on ResNet-50-C4. This highlights the division of labor in our approach, which largely separates classification and segmentation.

RoIAlign: An evaluation of our proposed RoIAlign layer is shown in Table 2c. For this experiment we use the ResNet50-C4 backbone, which has stride 16. RoIAlign improves AP by about 3 points over RoIPool, with much of the gain coming at high IoU (AP75). RoIAlign is insensitive to max/average pool; we use average in the rest of the paper.

RoIAlign: The evaluation of the proposed RoIAlign layer is shown in Table 2c. In this experiment we use the ResNet50-C4 backbone with 16 strides. RoIAlign improves AP by about 3 points over RoIPool, and the high gain occurs at high IoU (AP75). RoIAlign is insensitive to max / average pooling. We use the average in the rest of the paper.

Additionally, we compare with RoIWarp proposed in MNC ²³⁸ that also adopt bilinear sampling. As discussed in §3, RoIWarp still quantizes the RoI, losing alignment with the input. As can be seen in Table 2c, RoIWarp performs on par with RoIPool and much worse than RoIAlign. This highlights that proper alignment is key.

We also compare with RoIWarp, proposed in MNC ²³⁹, which adopted bilinear sampling. As discussed in §3, RoIWarp still quantizes the RoI and loses alignment with the input. As Table 2c shows, RoIWarp performs on par with RoIPool and much worse than RoIAlign. This highlights that proper alignment is the key.

We also evaluate RoIAlign with a ResNet-50-C5 backbone, which has an even larger stride of 32 pixels. We use the same head as in Figure 4 (right), as the res5 head is not applicable. Table 2d shows that RoIAlign improves mask AP by a massive 7.3 points, and mask AP75 by 10.5 points (50% relative improvement). Moreover, we note that with RoIAlign, using stride-32 C5 features (30.9 AP) is more accurate than using stride-16 C4 features (30.3 AP, Table 2c). RoIAlign largely resolves the long-standing challenge of using large-stride features for detection and segmentation.

We also evaluate RoIAlign using a ResNet-50-C5 backbone with a 32-pixel stride. Since the res5 head cannot be applied, we use the same head as Figure 4 (right). Table 2d shows that RoIAlign greatly improves mask AP by 7.3 points and improves mask AP75 by 10.5 points (50% relative improvement). Also, with RoIAlign, using stride-32 C5 features (30.9 AP) is more accurate than using stride-16 C4 features (30.3 AP, Table 2c). RoIAlign largely solves the long-standing challenge of using large-stride features for detection and segmentation.

Finally, RoIAlign shows a gain of 1.5 mask AP and 0.5 box AP when used with FPN, which has finer multi-level strides. For keypoint detection that requires finer alignment, RoIAlign shows large gains even with FPN (Table 6).

Finally, RoIAlign shows a gain of 1.5 mask AP and 0.5 box AP when used with FPN. For keypoint detection, which needs more precise alignment, RoIAlign shows large gains even with FPN (Table 6).

Mask Branch: Segmentation is a pixel-to-pixel task and we exploit the spatial layout of masks by using an FCN. In Table 2e, we compare multi-layer perceptrons (MLP) and FCNs, using a ResNet-50-FPN backbone. Using FCNs gives a 2.1 mask AP gain over MLPs. We note that we choose this backbone so that the conv layers of the FCN head are not pre-trained, for a fair comparison with MLP

Mask branch: Segmentation is a pixel-to-pixel task, and we use an FCN to exploit the spatial layout of the mask. Table 2e compares MLP (Multi-Layer Perceptron) and FCN using a ResNet-50-FPN backbone. Using FCN gives a 2.1 mask AP gain over MLP. For a fair comparison with MLP, we choose this backbone so that the conversion layers of the FCN head are not pre-trained

We compare Mask R-CNN to the state-of-the-art COCO bounding-box object detection in Table 3. For this result, even though the full Mask R-CNN model is trained, only the classification and box outputs are used at inference (the mask output is ignored). Mask R-CNN using ResNet-101- FPN outperforms the base variants of all previous state-ofthe-art models, including the single-model variant of GRMI ²⁴⁰, the winner of the COCO 2016 Detection Challenge. Using ResNeXt-101-FPN, Mask R-CNN further improves results, with a margin of 3.0 points box AP over the best previous single model entry from ²⁴¹ (which used Inception-ResNet-v2-TDM).

We compare Mask R-CNN with the latest COCO bounding box object detection in Table 3.For this result, even though the full Mask R-CNN model was trained, the classification and box outputs are used for inference (the mask output is ignored). Mask R-CNN using ResNet-101-FPN outperforms the base variants of every previous state-of-the-art model, including the single-model variant ²⁴² of GRMI, the winner of the COCO 2016 Detection Challenge. Using ResNeXt-101-FPN, Mask R-CNN improves the results further, with a margin of 3.0 points box AP over the oldest previous single-model entry of ²⁴³ (which used Inception-ResNet-v2-TDM).

As a further comparison, we trained a version of Mask R-CNN but without the mask branch, denoted by “Faster R-CNN, RoIAlign” in Table 3. This model performs better than the model presented in ²⁴⁴ due to RoIAlign. On the other hand, it is 0.9 points box AP lower than Mask R-CNN. This gap of Mask R-CNN on box detection is therefore due solely to the benefits of multi-task training.

For further comparison, we trained a version of Mask R-CNN, but without the mask branch, and marked it "Faster R-CNN, RoIAlign" in Table 3.Because of RoIAlign, this model performs better than the model presented in ²⁴⁵. On the other hand, its box AP is 0.9 points lower than Mask R-CNN. This difference of Mask R-CNN in box detection is entirely due to the benefits of multi-task training.

Lastly, we note that Mask R-CNN attains a small gap between its mask and box AP: e.g., 2.7 points between 37.1 (mask, Table 1) and 39.8 (box, Table 3). This indicates that our approach largely closes the gap between object detection and the more challenging instance segmentation task.

Finally, Mask R-CNN gets a small gap between mask and box AP (e.g.: 2.7 points between 37.1 (mask, Table 1) and 39.8 (box, Table 3)). This shows that our approach greatly narrows the gap between object detection and the trickier instance segmentation task.

Inference: We train a ResNet-101-FPN model that shares features between the RPN and Mask R-CNN stages, following the 4-step training of Faster R-CNN ²⁴⁶. This model runs at 195ms per image on an Nvidia Tesla M40 GPU (plus 15ms CPU time resizing the outputs to the original resolution), and achieves statistically the same mask AP as the unshared one. We also report that the ResNet-101-C4 variant takes ∼400ms as it has a heavier box head (Figure 4), so we do not recommend using the C4 variant in practice.

Inference: Following the 4-step training of Faster R-CNN ²⁴⁷, we train a ResNet-101-FPN model that shares features between the RPN and Mask R-CNN stages. This model runs at 195ms per image on an Nvidia Tesla M40 GPU (more than 15ms of CPU time to resize the output to the original resolution) and statistically achieves the same mask AP as the unshared one. Also, the ResNet-101-C4 variant has a heavier box head and so takes ~ 400ms, so in practice it is better not to use the C4 variant.

Mask R-CNN is fast, but our design is not optimized for speed, and a better speed / accuracy trade-off can be obtained (e.g.: by changing the image size and proposal number, which is outside the scope of this paper).

Although Mask R-CNN is fast, we note that our design is not optimized for speed, and better speed/accuracy tradeoffs could be achieved ²⁴⁸, e.g., by varying image sizes and proposal numbers, which is beyond the scope of this paper.

Training: Mask R-CNN is also fast to train. Training with ResNet-50-FPN on COCO trainval35k takes 32 hours in our synchronized 8-GPU implementation (0.72s per 16- image mini-batch), and 44 hours with ResNet-101-FPN. In fact, fast prototyping can be completed in less than one day when training on the train set. We hope such rapid training will remove a major hurdle in this area and encourage more people to perform research on this challenging topic.

Training: Mask R-CNN also trains quickly. Training with ResNet-50-FPN on COCO trainval35k takes 32 hours in the synchronized 8-GPU implementation (0.72 per 32-image mini-batch), and 44 hours with ResNet-101-FPN. In fact, when training on the train set, fast prototyping can be finished in a single day. We hope such fast training removes a major obstacle in this field and encourages more people to do research on this challenging topic.

Our framework can easily be extended to human pose estimation. We model a keypoint’s location as a one-hot mask, and adopt Mask R-CNN to predict K masks, one for each of K keypoint types (e.g., left shoulder, right elbow). This task helps demonstrate the flexibility of Mask R-CNN

Our framework can easily be extended to human pose estimation. We model the location of a keypoint as a one-hot mask and adopt Mask R-CNN to predict K masks, one for each of the K keypoint types (e.g.: left shoulder, right elbow). This task helps prove the flexibility of Mask R-CNN.

We note that minimal domain knowledge for human pose is exploited by our system, as the experiments are mainly to demonstrate the generality of the Mask R-CNN framework. We expect that domain knowledge (e.g., modeling structures ²⁴⁹) will be complementary to our simple approach.

Because the experiments mainly show the generality of the Mask R-CNN framework, the system uses only minimal domain knowledge about human pose. We expect domain knowledge (for example, modeling structures ²⁵⁰) to complement our simple approach.

Implementation Details: We make minor modifications to the segmentation system when adapting it for keypoints. For each of the K keypoints of an instance, the training target is a one-hot m × m binary mask where only a single pixel is labeled as foreground. During training, for each visible ground-truth keypoint, we minimize the cross-entropy loss over an m2 -way softmax output (which encourages a single point to be detected). We note that as in instance segmentation, the K keypoints are still treated independently.

Implementation details: We slightly modify the segmentation system when adapting it to keypoints. For each of the K keypoints of an instance, the training target is a one-hot m × m binary mask in which only a single pixel is labeled as foreground. During training, for each visible ground-truth keypoint, we minimize the cross-entropy loss over the m2-way softmax output (so that a single point is detected). As in instance segmentation, the K keypoints are still handled independently.

We adopt the ResNet-FPN variant, and the keypoint head architecture is similar to that in Figure 4 (right). The keypoint head consists of a stack of eight 3×3 512-d conv layers, followed by a deconv layer and 2× bilinear upscaling, producing an output resolution of 56×56. We found that a relatively high resolution output (compared to masks) is required for keypoint-level localization accuracy.

We adopted the ResNet-FPN variant, and the keypoint head architecture is similar to Figure 4 (right). The keypoint head consists of a stack of 8 3x3 512-d conv layers, plus a deconv layer and 2x bilinear upscaling, producing an output resolution of 56x56. Keypoint-level localization accuracy requires a relatively high-resolution output compared with masks.

Models are trained on all COCO trainval35k images that contain annotated keypoints. To reduce overfitting, as this training set is smaller, we train using image scales randomly sampled from [640, 800] pixels; inference is on a single scale of 800 pixels. We train for 90k iterations, starting from a learning rate of 0.02 and reducing it by 10 at 60k and 80k iterations. We use bounding-box NMS with a threshold of 0.5. Other details are identical as in §3.1.

The model is trained on all COCO trainval35k images that contain annotated keypoints. To reduce overfitting, the smaller this training set is, we train using image scales randomly sampled from [640, 800] pixels. Inference is at a single scale of 800 pixels. Starting at a learning rate of 0.02 and reducing it by 10 at 60k and 80k iterations, we train for 90k iterations. We use bounding box NMS with a threshold of 0.5. Other details are the same as in §3.1.

Main Results and Ablations: We evaluate the person keypoint AP (APkp) and experiment with a ResNet-50-FPN backbone; more backbones will be studied in the appendix. Table 4 shows that our result (62.7 APkp) is 0.9 points higher than the COCO 2016 keypoint detection winner ²⁵¹ that uses a multi-stage processing pipeline (see caption of Table 4). Our method is considerably simpler and faster.

Main results and removals: We evaluate the individual keypoint AP (APkp) and experiment with a ResNet-50-FPN backbone. More backbones will be studied in the appendix. Table 4 shows that our result (62.7 APkp) is 0.9 points higher than the COCO 2016 keypoint detection winner ²⁵², which uses a multi-stage processing pipeline (see the Table 4 caption). Our method is much simpler and faster.

More importantly, we have a unified model that can simultaneously predict boxes, segments, and keypoints while running at 5 fps. Adding a segment branch (for the person category) improves the APkp to 63.1 (Table 4) on test-dev. More ablations of multi-task learning on minival are in Table 5. Adding the mask branch to the box-only (i.e., Faster R-CNN) or keypoint-only versions consistently improves these tasks. However, adding the keypoint branch reduces the box/mask AP slightly, suggesting that while keypoint detection benefits from multitask training, it does not in turn help the other tasks. Nevertheless, learning all three tasks jointly enables a unified system to efficiently predict all outputs simultaneously (Figure 7).

More importantly, there is a unified model that can predict boxes, segments, and keypoints at the same time while running at 5fps. Adding a segment branch for the individual category improves APkp on test-dev to 63.1 (Table 4). More ablations of multi-task learning on minival are in Table 5. Adding the mask branch to the box-only (that is, Faster R-CNN) or keypoint-only versions consistently improves these tasks. However, adding the keypoint branch slightly reduces box / mask AP, so keypoint detection benefits from multi-task training but does not help the other tasks. Nevertheless, learning all three tasks lets the unified system predict all outputs efficiently at the same time (Figure 7).

We also investigate the effect of RoIAlign on keypoint detection (Table 6). Though this ResNet-50-FPN backbone has finer strides (e.g., 4 pixels on the finest level), RoIAlign still shows significant improvement over RoIPool and increases APkp by 4.4 points. This is because keypoint detections are more sensitive to localization accuracy. This again indicates that alignment is essential for pixel-level localization, including masks and keypoints.

We also examine the effect of RoIAlign on keypoint detection (Table 6). This ResNet-50-FPN backbone has finer strides (e.g.: 4 pixels at the highest level), but RoIAlign still shows a considerable improvement over RoIPool, raising APkp by 4.4 points. This is because keypoint detection is more sensitive to localization accuracy. This shows that alignment is essential for pixel-level localization, including masks and keypoints.

Given the effectiveness of Mask R-CNN for extracting object bounding boxes, masks, and keypoints, we expect it be an effective framework for other instance-level tasks.

Given the effectiveness of Mask R-CNN in extracting object bounding boxes, masks, and keypoints, we expect it to be an effective framework for other instance-level tasks.

We further report instance segmentation results on the Cityscapes ²⁵³ dataset. This dataset has fine annotations for 2975 train, 500 val, and 1525 test images. It has 20k coarse training images without instance annotations, which we do not use. All images are 2048×1024 pixels. The instance segmentation task involves 8 object categories, whose numbers of instances on the fine training set are:

We additionally report instance segmentation results on the Cityscapes ²⁵⁴ dataset. This dataset has annotations for 2975 train, 500val, and 1525 test images. There are 20k coarse training images without instance annotations, which we do not use. All images are 2048 × 1024 pixels. The instance segmentation task includes 8 object categories, and the number of instances in the fine training set is as follows

Instance segmentation performance on this task is measured by the COCO-style mask AP (averaged over IoU thresholds); AP50 (i.e., mask AP at an IoU of 0.5) is also reported.

Instance segmentation performance on this task is measured with COCO-style mask AP (averaged over IoU thresholds). AP50 (that is, mask AP at an IoU of 0.5) is also reported.

Implementation: We apply our Mask R-CNN models with the ResNet-FPN-50 backbone; we found the 101-layer counterpart performs similarly due to the small dataset size. We train with image scale (shorter side) randomly sampled from [800, 1024], which reduces overfitting; inference is on a single scale of 1024 pixels. We use a mini-batch size of 1 image per GPU (so 8 on 8 GPUs) and train the model for 24k iterations, starting from a learning rate of 0.01 and reducing it to 0.001 at 18k iterations. It takes ∼4 hours of training on a single 8-GPU machine under this setting.

Implementation: We apply the Mask R-CNN model with the ResNet-FPN-50 backbone. We found that the 101-layer counterpart performs similarly because of the small dataset size. We reduce overfitting by training with image scales (shorter side) randomly sampled from [800, 1024]. Inference is at a single scale of 1024 pixels. We use a mini-batch size of 1 image per GPU (8 GPU 8) and train the model for 24k iterations, starting at a learning rate of 0.01 and going to 0.001 at 18k iterations. With this setting, training takes about 4 hours on a single 8-GPU system.

Results: Table 7 compares our results to the state of the art on the val and test sets. Without using the coarse training set, our method achieves 26.2 AP on test, which is over 30% relative improvement over the previous best entry (DIN ²⁵⁵), and is also better than the concurrent work of SGN’s 25.0 ²⁵⁶. Both DIN and SGN use fine + coarse data. Compared to the best entry using fine data only (17.4 AP), we achieve a ∼50% improvement.

Results: Table 7 compares the results with the state of the art on the val and test sets. Without using the approximate training set, our method reaches 26.2 AP on the test, a relative improvement of more than 30% over the previous best entry (DIN ²⁵⁷), and it is better than the concurrent work of SGN's 25.0 ²⁵⁸. DIN and SGN both use fine + coarse data. Compared with the best entry that uses only fine data (17.4 AP), it improved by ~ 50%.

For the person and car categories, the Cityscapes dataset exhibits a large number of within-category overlapping instances (on average 6 people and 9 cars per image). We argue that within-category overlap is a core difficulty of instance segmentation. Our method shows massive improvement on these two categories over the other best entries (relative ∼40% improvement on person from 21.8 to 30.5 and ∼20% improvement on car from 39.4 to 46.9), even though our method does not exploit the coarse data.

For the individual and car categories, the Cityscapes dataset shows a large number of instances that overlap within a category (on average 6 people and 9 cars per image). We argue that within-category overlap is the core difficulty of instance segmentation. Even though our method does not use the approximate data, it shows large improvements in these two categories over the other best entries (~ 40% improvement for person from 21.8 to 30.5, and ~ 20% improvement for car from 39.4 to 46.9).

A main challenge of the Cityscapes dataset is training models in a low-data regime, particularly for the categories of truck, bus, and train, which have about 200-500 training samples each. To partially remedy this issue, we further report a result using COCO pre-training. To do this, we initialize the corresponding 7 categories in Cityscapes from a pre-trained COCO Mask R-CNN model (rider being randomly initialized). We fine-tune this model for 4k iterations in which the learning rate is reduced at 3k iterations, which takes ∼1 hour for training given the COCO model.

The main challenge of the Cityscapes dataset is training models in a low-data region, especially for the truck, bus, and train categories, which have 200,500 training samples. To partly solve this problem, we additionally report results using COCO pre-training. To do this, we initialize the corresponding 7 categories of Cityscapes from a pre-trained COCO Mask R-CNN model (rider is randomly initialized). We fine-tune this model for 4k iterations, with the learning rate reduced at 3k iterations. Given the COCO model, training takes about 1 hour.

The COCO pre-trained Mask R-CNN model achieves 32.0 AP on test, almost a 6 point improvement over the fine-only counterpart. This indicates the important role the amount of training data plays. It also suggests that methods on Cityscapes might be influenced by their lowshot learning performance. We show that using COCO pretraining is an effective strategy on this dataset.

The COCO pre-trained Mask R-CNN model reaches 32.0 AP on the test, almost 6 points better than the fine-only product. This shows that the amount of training data plays an important role. It also suggests that methods on Cityscapes may be affected by low learning performance. We show that using COCO pre-training is an effective strategy for this dataset.

Finally, we observed a bias between the val and test AP, as is also observed from the results of^259260261. We found that this bias is mainly caused by the truck, bus, and train categories, with the fine-only model having val/test AP of 28.8/22.8, 53.5/32.2, and 33.0/18.6, respectively. This suggests that there is a domain shift on these categories, which also have little training data. COCO pre-training helps to improve results the most on these categories; however, the domain shift persists with 38.0/30.1, 57.5/40.9, and 41.2/30.9 val/test AP, respectively. Note that for the person and car categories we do not see any such bias (val/test AP are within ±1 point). Example results on Cityscapes are shown in Figure 8.

Finally, as can be seen in the results of^262263264, we observed a skew between val and test AP. We found that this bias is caused mainly by the truck, bus, and train categories. The val / test AP of the fine model is 28.8 / 22.8, 53.5 / 32.2, and 33.0 / 18.6 respectively. This indicates a domain shift in which there is little training material for these categories. COCO pre-training helps improve results the most in these categories. However, the domain shift persists, with val / test AP of 38.0 / 30.1, 57.5 / 40.9, and 41.2 / 30.9 respectively. For the individual and car categories, no such deviation appears (val / test AP within ± 1 point). Example results on Cityscapes are shown in Figure 8.

As a general framework, Mask R-CNN is compatible with complementary techniques developed for detection/segmentation, including improvements made to Fast/Faster R-CNN and FCNs. In this appendix we describe some techniques that improve over our original results. Thanks to its generality and flexibility, Mask R-CNN was used as the framework by the three winning teams in the COCO 2017 instance segmentation competition, which all significantly outperformed the previous state of the art.

As a general framework, Mask R-CNN is compatible with complementary techniques developed for detection / segmentation, including improvements to Fast / Faster R-. CNN and FCN. This appendix describes several techniques that improve on the original results. Thanks to its generality and flexibility, Mask R-CNN was used as the framework by 3 winning teams in the COCO 2017 instance segmentation competition, and this outperformed all of the previous state of the art.

We report some enhanced results of Mask R-CNN in Table 8. Overall, the improvements increase mask AP 5.1 points (from 36.7 to 41.8) and box AP 7.7 points (from 39.6 to 47.3). Each model improvement increases both mask AP and box AP consistently, showing good generalization of the Mask R-CNN framework. We detail the improvements next. These results, along with future updates, can be reproduced by our released code at https://github.com/facebookresearch/Detectron, and can serve as higher baselines for future research.

Some enhanced results of Mask R-CNN are reported in Table 8. Overall, the improvements are mask AP 5.1 points (from 36.7 to 41.8) and box AP 7.7 points (increase from 39.6 to 47.3). ). Each model improvement consistently increases mask AP and box AP, showing the generalization of the Mask R-CNN framework. The following improvements are described in detail. These results, together with future updates, can be reproduced with the released code at https://github.com/facebookresearch/Detectron, and they can become higher baselines for future research.

Updated baseline: We start with an updated baseline with a different set of hyper-parameters. We lengthen the training to 180k iterations, in which the learning rate is reduced by 10 at 120k and 160k iterations. We also change the NMS threshold to 0.5 (from a default value of 0.3). The updated baseline has 37.0 mask AP and 40.5 box AP.

Updated baseline: We start with a baseline updated with a different set of hyperparameters. Training is extended to 180k iterations, and the learning rate is reduced by 10 at 120k and 160k iterations. We also change the NMS threshold to 0.5 (from the default of 0.3). The updated baseline has 37.0 mask AP and 40.5 box AP.

End-to-end training: All previous results used stagewise training, i.e., training RPN as the first stage and Mask R-CNN as the second. Following ²⁶⁵, we evaluate endto-end (‘e2e’) training that jointly trains RPN and Mask RCNN. We adopt the ‘approximate’ version in ²⁶⁶ that only computes partial gradients in the RoIAlign layer by ignoring the gradient w.r.t. RoI coordinates. Table 8 shows that e2e training improves mask AP by 0.6 and box AP by 1.2.

End-to-end training: All previous results used stage-by-stage training, that is, training RPN as the first stage and Mask R-CNN as the second stage. Following ²⁶⁷, we evaluate end-to-end ('e2e') training that trains RPN and Mask RCNN jointly. We adopt the 'approximate' version of ²⁶⁸, which computes only partial gradients in the RoIAlign layer by ignoring the gradient wrt RoI coordinates. Table 8 shows that e2e training improves mask AP to 0.6 and box AP to 1.2.

ImageNet-5k pre-training: Following ²⁶⁹, we experiment with models pre-trained on a 5k-class subset of ImageNet (in contrast to the standard 1k-class subset). This 5× increase in pre-training data improves both mask and box 1 AP. As a reference, ²⁷⁰ used ∼250× more images (300M) and reported a 2-3 box AP improvement on their baselines.

ImageNet-5k pre-training: Following ²⁷¹, we experiment with models pre-trained on a 5k-class subset of ImageNet (in contrast to the standard 1k-class subset). A 5x increase in pre-training data improves both mask and box 1 AP. For reference, ²⁷² used about 250x more images (300M) and reported a 2-3 box AP improvement over the baseline.

Train-time augmentation: Scale augmentation at train time further improves results. During training, we randomly sample a scale from [640, 800] pixels and we increase the number of iterations to 260k (with the learning rate reduced by 10 at 200k and 240k iterations). Train-time augmentation improves mask AP by 0.6 and box AP by 0.8.

Training-time augmentation: Scale augmentation at training time improves the results further. During training, we randomly sample a scale from [640, 800] pixels and increase the number of iterations to 260k (the learning rate decreases by 10 at 200k and 240k iterations). Train-time augmentation improves mask AP to 0.6 and box AP to 0.8.

Model architecture: By upgrading the 101-layer ResNeXt to its 152-layer counterpart ²⁷³, we observe an increase of 0.5 mask AP and 0.6 box AP. This shows a deeper model can still improve results on COCO.

Model architecture: By upgrading the 101-layer ResNeXt to the 152-layer counterpart ²⁷⁴, we can observe an increase of 0.5 mask AP and 0.6 box AP. This shows that a deeper model can still improve results on COCO.

Using the recently proposed non-local (NL) model ²⁷⁵, we achieve 40.3 mask AP and 45.0 box AP. This result is without test-time augmentation, and the method runs at 3fps on an Nvidia Tesla P100 GPU at test time.

Using the recently proposed non-local (NL) model ²⁷⁶, we reached 40.3 mask AP and 45.0 box AP. This result runs at 3fps on an Nvidia Tesla P100 GPU at test time, without increasing test time.

Test-time augmentation: We combine the model results evaluated using scales of [400, 1200] pixels with a step of 100 and on their horizontal flips. This gives us a singlemodel result of 41.8 mask AP and 47.3 box AP.

Test-time augmentation: We combine the model results evaluated using scales of [400, 1200] pixels with steps of 100 and horizontal flips. This gives us a single-model result of 41.8 mask AP and 47.3 box AP.

The above result is the foundation of our submission to the COCO 2017 competition (which also used an ensemble, not discussed here). The first three winning teams for the instance segmentation task were all reportedly based on an extension of the Mask R-CNN framework.

The result above is what we submitted to the COCO 2017 competition (using an ensemble not discussed here). The first 3 winning teams in the instance segmentation task were all reported to be based on extensions of the Mask R-CNN framework.

We report enhanced results of keypoint detection in Table 9. As an updated baseline, we extend the training schedule to 130k iterations in which the learning rate is reduced by 10 at 100k and 120k iterations. This improves APkp by about 1 point. Replacing ResNet-50 with ResNet-101 and ResNeXt-101 increases APkp to 66.1 and 67.3, respectively.

Table 9 reports enhanced keypoint detection results. As the updated baseline, we extend the training schedule to 130k iterations, with the learning rate reduced by 10 at 100k and 120k iterations. This improves APkp by about 1 point. Replacing ResNet-50 with ResNet-101 and ResNeXt-101 raises APkp to 66.1 and 67.3 respectively.

With a recent method called data distillation ²⁷⁷, we are able to exploit the additional 120k unlabeled images provided by COCO. In brief, data distillation is a self-training strategy that uses a model trained on labeled data to predict annotations on unlabeled images, and in turn updates the model with these new annotations. Mask R-CNN provides an effective framework for such a self-training strategy. With data distillation, Mask R-CNN APkp improve by 1.8 points to 69.1. We observe that Mask R-CNN can benefit from extra data, even if that data is unlabeled.

Using a recent method called data distillation ²⁷⁸, we can take advantage of the additional 120k unlabeled images that COCO provides. Put simply, data distillation is a self-training strategy that uses a model trained on labeled data to predict annotations for unlabeled images, and updates the model with these new annotations. Mask R-CNN provides an effective framework for such a self-training strategy. With data distillation, Mask R-CNN APkp improved by 1.8 points to 69.1. Even when the data is unlabeled, Mask R-CNN can benefit from additional data.

By using the same test-time augmentation as used for instance segmentation, we further boost APkp to 70.4.

Using the same test-time feature augmentation as used for, for example, segmentation, we further improve APkp to 70.4.

Acknowledgements: We would like to acknowledge Ilija Radosavovic for contributions to code release and enhanced results, and the Caffe2 team for engineering support.

Acknowledgements: For Ilija Radosavovic contributing to the code release and the improved results, and the Caffe2 team, we thank them for engineering support.