RTMDet
lastmod 2026-01-18created 2024-03-0610 edits
RTMDet: An Empirical Study of Designing Real-Time Object Detectors
Abstract
- Does real-time object detection better than the YOLO series
- Easily extensible to tasks such as instance segmentation
- Compared with detection tasks, segmentation tasks lose performance when the output feature resolution is too small.
- Thanks to the structure below, proposes an architecture whose backbone and neck have compatible capacity (a structure designed so that no part is limited by the performance or capacity of another)
- A basic building block made of large-kernel depth-wise convolutions (the backbone and neck both use the same basic building block)
- The concept of depth-wise convolution and why it's used: https://velog.io/@hsbc/depth-wise-separable-convolution
- (In the dynamic label assignment problem,) improves accuracy by introducing soft labels when computing the matching cost
Simple Example
See MMDetection#Simple Example.
Internal test results
- Measured with mmdetection
- NVIDIA GeForce RTX 2080 Ti
- rtmdet-ins_x_8xb16-300e_coco about 0.056 s (56 ms)
- rtmdet-ins_s_8xb32-300e_coco about 0.023 s (23 ms)
- Using the rtmdet-ins_x_8xb16-300e_coco_dongkuk_rolling_plate_cam1_2nd.py model.
- 3840x2160 (4K Original) - 0.0161s ~ 0.0176s (about 17ms)
- 1660x1926 (ROI Crop) - 0.0159s ~ 0.0182s (about 18ms) - seems safe to assume it's simply not affected by image size.
Problem: staircase artifacts
- See Dongkuk Dangjin Rolling Mill#RTMDet Camber camera result review.
| |
In the image above, the boundary fluctuates as if stepped (or like a SIN graph).
| WARNING |
| The content below is an answer from ChatGPT Deep Research. |
Why staircase artifacts occur in RTMDet: the effect of the upsampling structure
RTMDet (Real-Time Multi-Scale Detection) is an object detection (Object Detection) model, fundamentally based on the FCOS (Anchor-Free Detection) architecture. That is, it is not a segmentation model that predicts per pixel; it uses a box-based detection-then-mask-generation approach. To achieve real-time performance it uses multi-resolution feature maps, and upsampling (Up-sampling) occurs in this process.
The upsampling process and staircase artifacts
In general, detection-based segmentation models (including the RTMDet family) go through the following upsampling steps:
- Backbone (Backbone) downsampling
- Typically uses a backbone such as ResNet or CSPNet to shrink the image to 1/16–1/32 of its size.
- High-resolution information is lost.
- Combining multi-resolution features via FPN/PAFPN
- Combines feature maps of multiple sizes to supplement information.
- However, low-resolution features must be upsampled to restore high resolution.
- Mask generation via upsampling:
- Rather than the Fully Convolutional approach of the FCN family, masks are basically generated using ROI Align or Deformable Convolution.
- As the mask is upsampled (1/32 -> 1/16 -> 1/8 -> original size), the gaps between pixels are filled by interpolation, producing staircase artifacts.
- Effect of the interpolation method:
- Upsampling typically uses Bilinear Interpolation or Deconvolution (Transpose Convolution).
- However, Bilinear Interpolation cannot connect pixels smoothly, blurring boundaries or causing staircase artifacts.
- Transpose Convolution (Deconv, Fractionally-strided Convolution) can produce Checkerboard Artifacts (checkerboard patterns, block noise).
RTMDet vs. typical segmentation models
Dedicated segmentation models use structures that preserve high resolution to reduce staircase artifacts, but RTMDet generates masks after object detection, so it doesn't preserve enough high-resolution information.
| Model | Features | Upsampling method | Likelihood of staircase artifacts |
| RTMDet-Seg | Detection-based | Bilinear, Deconv | High |
| Fast, lightweight segmentation | Uses a Detail Branch | Low | |
| Adds a boundary branch | Boundary-aware Fusion | Minimized | |
| Dual-resolution structure | Keeps a High-Resolution Path | Almost none |
In particular, PIDNet and DDRNet reduce staircase artifacts by using almost no upsampling or by training a separate boundary branch, so they give much smoother boundaries than RTMDet.
Ways to reduce staircase artifacts in the RTMDet family
- Hardware upsampling correction
- Post-processing correction using bicubic upsampling (using PyTorch
F.interpolate(mode='bicubic')). - Noise correction after upsampling with OpenCV's cv2.GaussianBlur and similar.
- Changing the model itself
- Change the model to keep a High-Resolution Path instead of Deformable Convolution (add a DDRNet-style structure).
- Add an Extra Boundary Branch for boundary-based separation (PIDNet style).
- Replacing the model
- To solve the upsampling problem while keeping real-time processing, the most effective option is to use a model that preserves a high-resolution structure, such as STDC, PIDNet, or DDRNet.
Conclusion
The main cause of staircase artifacts in RTMDet is the process of restoring resolution during the upsampling stage. Typical segmentation models minimize upsampling or compensate with a separate structure, but RTMDet interpolates masks after object detection, which causes this problem. Using PIDNet or DDRNet can reduce staircase artifacts while keeping real-time speed.
Related pages
Further related content will be organized in Segmentation Artifacts.
Documentation
- [2212.07784] RTMDet - An Empirical Study of Designing Real-Time Object Detectors
- https://arxiv.org/abs/2212.07784