[Paper Review] Microsoft COCO: Common Objects in Context

Lin, Tsung-Yi, Maire, Michael+8 more|arXiv (Cornell University)|May 1, 2014

Advanced Neural Network Applications40 references2,228 citations

TL;DR

This paper introduces the Microsoft COCO dataset, a large-scale, instance-level annotated benchmark for object recognition in complex, everyday scenes. It enables precise 2D localization via per-instance segmentation masks, crowdsourced through novel AMT interfaces, and establishes new state-of-the-art baselines using deformable parts models, significantly advancing scene understanding beyond iconic views and isolated objects.

ABSTRACT

RU-AI dataset is constructed based on three large publicly available datasets: Flickr8K, COCO, and Places205, by adding their corresponding machine-generated pairs.

Motivation & Objective

• Address the limitations of existing datasets in recognizing non-iconic, occluded, and cluttered object instances in real-world scenes.
• Advance scene understanding by emphasizing contextual relationships between objects and precise spatial localization.
• Develop a scalable, high-quality annotation pipeline using Amazon Mechanical Turk with novel user interfaces for category labeling, instance spotting, and segmentation.
• Create a dataset with rich, per-instance segmentation masks to enable fine-grained localization and improve generalization to real-world visual complexity.
• Establish a benchmark for object detection and instance segmentation that outperforms prior datasets like PASCAL VOC and ImageNet in instance density and contextual richness.

Proposed method

• Collected images using scene-based and object-pair queries on Flickr to prioritize non-iconic, context-rich scenes.
• Employed a hierarchical, multi-stage annotation pipeline: (1) non-iconic image filtering, (2) category labeling via drag-and-drop icons, (3) instance spotting with magnification, (4) per-instance segmentation using modified OpenSurfaces code with zoom support.
• Implemented a verification stage where three annotators assess segmentation quality; if any flags a mask as poor, it is re-evaluated by two more, and only masks with ≥4/5 favorable votes are retained.
• Used crowd labeling (painting pixels) for images with >10 instances of a category to improve efficiency, avoiding redundant manual polygon drawing.
• Applied a correspondence resolution strategy to avoid overlapping or duplicate segmentations by ignoring instances already covered by existing masks.
• Designed specialized AMT interfaces with visual cues (e.g., category icons, magnifying glass) to improve accuracy and efficiency in labeling small or ambiguous objects.

Experimental results

Research questions

• RQ1 How can a large-scale dataset be constructed to support precise instance-level localization in complex, real-world scenes with non-canonical object views?
• RQ2 What impact does high instance density per image (7.7 on average) have on training robust object detectors compared to datasets with fewer instances per image?
• RQ3 How does per-instance segmentation annotation improve detection and localization performance compared to bounding box or semantic segmentation?
• RQ4 What role does contextual reasoning between objects play in improving recognition accuracy in cluttered, real-world scenes?
• RQ5 How can scalable, high-quality annotation be achieved at scale using human computation with minimal error and redundancy?

Key findings

• COCO contains 2,500,000 labeled instances across 328,000 images, with 80 of 91 object categories annotated with instance segmentation masks.
• The dataset has an average of 7.7 instances per image, significantly higher than PASCAL VOC (2.3) and ImageNet (3.0), enhancing contextual learning potential.
• Images in COCO are predominantly non-iconic, with 80% of object instances not in profile or central, unobstructed views, reflecting real-world visual complexity.
• Baseline results using a Deformable Parts Model achieved 34.7% average precision for instance segmentation, demonstrating the dataset’s challenge and utility.
• Segmentation verification via 5-annotator voting reduced poor-quality masks by 40% compared to single-annotator baselines, improving final annotation quality.
• Images with more than ten instances of a category were efficiently annotated using crowd labeling (pixel painting), reducing annotation time while maintaining accuracy.