[Paper Review] Meticulous Object Segmentation

Chenglin Yang, Yilin Wang+4 more|arXiv (Cornell University)|Dec 13, 2020

Visual Attention and Saliency Detection49 references4 citations

TL;DR

This paper introduces Meticulous Object Segmentation (MOS), a new task focused on high-resolution (2k–4k) image segmentation of complex, finely detailed objects. The authors propose MeticulousNet, a network with a recursive, hierarchical point-wise refining decoder that improves boundary accuracy and mask quality, achieving state-of-the-art performance on the newly released MOS600 benchmark with a 78.16 MQ score.

ABSTRACT

Compared with common image segmentation tasks targeted at low-resolution images, higher resolution detailed image segmentation receives much less attention. In this paper, we propose and study a task named Meticulous Object Segmentation (MOS), which is focused on segmenting well-defined foreground objects with elaborate shapes in high resolution images (e.g. 2k - 4k). To this end, we propose the MeticulousNet which leverages a dedicated decoder to capture the object boundary details. Specifically, we design a Hierarchical Point-wise Refining (HierPR) block to better delineate object boundaries, and reformulate the decoding process as a recursive coarse to fine refinement of the object mask. To evaluate segmentation quality near object boundaries, we propose the Meticulosity Quality (MQ) score considering both the mask coverage and boundary precision. In addition, we collect a MOS benchmark dataset including 600 high quality images with complex objects. We provide comprehensive empirical evidence showing that MeticulousNet can reveal pixel-accurate segmentation boundaries and is superior to state-of-the-art methods for high resolution object segmentation tasks.

Motivation & Objective

• Address the lack of focus on high-resolution, fine-grained object segmentation in existing methods, which typically target low-resolution images.
• Formalize a new task, Meticulous Object Segmentation (MOS), for precise segmentation of complex, detailed foreground objects in 2k–4k images.
• Develop a new evaluation metric, Meticulosity Quality (MQ), that balances mask coverage and boundary precision to reflect real-world segmentation demands.
• Release the MOS600 benchmark dataset with 600 high-quality, complex-object images and pixel-accurate annotations for fair evaluation.
• Propose MeticulousNet as a strong baseline model with a novel recursive, coarse-to-fine refinement decoder for high-resolution boundary detail recovery.

Proposed method

• Design a Hierarchical Point-wise Refining (HierPR) block that enables localized, uncertainty-guided refinement of segmentation masks at multiple levels.
• Reformulate the decoding process as a recursive, coarse-to-fine refinement of the object mask, improving resolution and boundary accuracy iteratively.
• Integrate the decoder with a dual-path network: a low-resolution model (MeticulousNet_L) for initial prediction and a high-resolution model (MeticulousNet_H) for refinement.
• Use a shared MeticulousNet architecture across both low- and high-resolution stages to maintain consistency and enable end-to-end training.
• Apply the HierPR block with a configurable refinement area percentage (e.g., 10%) to control local refinement scope and avoid overfitting to noise.
• Enable model flexibility by demonstrating compatibility with lightweight encoders like MobileNetV3, reducing parameters by 20× with minimal performance drop.

Experimental results

Research questions

• RQ1 Can a segmentation model achieve superior performance on high-resolution images with complex, detailed object boundaries compared to existing SOTA methods?
• RQ2 How does the proposed recursive, coarse-to-fine refinement mechanism improve boundary accuracy and mask quality in high-resolution segmentation?
• RQ3 To what extent does the Hierarchical Point-wise Refining (HierPR) block enhance local boundary delineation without degrading global mask coverage?
• RQ4 How does the new Meticulosity Quality (MQ) metric compare to traditional metrics like IoU and mBA in capturing the trade-off between boundary precision and mask completeness?
• RQ5 Can the proposed MeticulousNet architecture generalize across different encoder backbones, including lightweight ones, while maintaining high performance on MOS tasks?

Key findings

• MeticulousNet achieves a 78.16 MQ score on the low-resolution version of MOS600, significantly outperforming baseline models and existing SOTA methods.
• The combination of HierPR and recursive refinement increases IoU by 2.01%, mBA by 1.93%, and MQ by 1.12% compared to the base network, demonstrating strong complementarity.
• Using a 10% refinement area in HierPR yields optimal performance, with higher areas (15%, 20%) leading to performance degradation due to overfitting to noise.
• The mobile variant of MeticulousNet (mobile_MeticulousNet) reduces parameters by 20× while incurring only a 0.73% drop in MQ, proving strong model adaptability.
• Failure cases are primarily due to overfitting to color variations and high-frequency textures, indicating a need for future work on robustness to local color changes.
• The MOS600 benchmark contains 600 high-resolution images with complex, intricate object shapes, providing a realistic and challenging testbed for MOS evaluation.