[Paper Review] Location-Sensitive Visual Recognition with Cross-IOU Loss

Kaiwen Duan, Lingxi Xie+4 more|arXiv (Cornell University)|Apr 11, 2021

Advanced Neural Network Applications27 citations

TL;DR

This paper introduces LSNet, a unified anchor-free framework for location-sensitive visual recognition (object detection, instance segmentation, and pose estimation) that predicts an anchor point plus a set of landmarks and optimizes with a novel cross-IOU loss to handle multi-scale objects. It achieves state-of-the-art anchor-free results on MS-COCO and demonstrates strong contour-based segmentation and pose estimation performance.

ABSTRACT

Object detection, instance segmentation, and pose estimation are popular visual recognition tasks which require localizing the object by internal or boundary landmarks. This paper summarizes these tasks as location-sensitive visual recognition and proposes a unified solution named location-sensitive network (LSNet). Based on a deep neural network as the backbone, LSNet predicts an anchor point and a set of landmarks which together define the shape of the target object. The key to optimizing the LSNet lies in the ability of fitting various scales, for which we design a novel loss function named cross-IOU loss that computes the cross-IOU of each anchor point-landmark pair to approximate the global IOU between the prediction and ground-truth. The flexibly located and accurately predicted landmarks also enable LSNet to incorporate richer contextual information for visual recognition. Evaluated on the MS-COCO dataset, LSNet set the new state-of-the-art accuracy for anchor-free object detection (a 53.5% box AP) and instance segmentation (a 40.2% mask AP), and shows promising performance in detecting multi-scale human poses. Code is available at https://github.com/Duankaiwen/LSNet

Motivation & Objective

• Motivate a unified location-sensitive formulation for detection, segmentation, and pose estimation.
• Develop LSNet that uses an anchor point and landmarks to define object geometry.
• Design a cross-IOU loss to balance global IOU accuracy with local landmark alignment.
• Enhance features via Pyramid Deformable Convolution around landmarks to improve recognition.

Proposed method

• Predict an anchor point and N landmarks on top of a backbone to define object geometry.
• Use cross-IOU loss to supervise landmark offsets by approximating global IOU through offset vectors.
• Apply Pyramid Deformable Convolution (Pyramid-DCN) around predicted landmarks to extract discriminative features.
• Train with a two-stage pipeline: anchor point + landmarks, then refine with DCN-based features for classification and localization.
• Integrate multi-scale features via FPN and ATSS-based assignment for robust localization.

Experimental results

Research questions

• RQ1 Can a unified anchor-free framework jointly handle object detection, instance segmentation, and pose estimation by predicting an anchor and landmarks?
• RQ2 Does the cross-IOU loss provide robust supervision for objects at multiple scales and varying shapes?
• RQ3 Do landmark-based features and Pyramid DCN improve localization and recognition across tasks?
• RQ4 How does LSNet perform against state-of-the-art anchor-free and contour-based methods on MS-COCO?
• RQ5 Is LSNet capable of effective multi-scale pose estimation without heatmaps?

Key findings

• LSNet achieves competitive to state-of-the-art results for anchor-free object detection on MS-COCO with high box AP when using stronger backbones and multi-scale testing.
• LSNet achieves state-of-the-art contour-based instance segmentation performance on COCO with single-scale and multi-scale testing.
• Cross-IOU loss provides improved performance over smooth-L1 and non-IOU variants for extreme bounding box regression and supports landmark-based supervision.
• Landmark features, including extreme points, and Pyramid-DCN features improve detection precision and localization.
• LSNet demonstrates competitive pose estimation results using regression without heatmaps, and scales well to multi-scale human poses.