[Paper Review] Polarity Loss for Zero-shot Object Detection

Shafin Rahman, Salman Khan+1 more|arXiv (Cornell University)|Nov 22, 2018

Multimodal Machine Learning Applications53 references45 citations

TL;DR

The paper introduces Polarity Loss to improve visual-semantic alignment for zero-shot object detection, incorporating vocabulary-aware metric learning and a single-stage RetinaNet-based detector to detect unseen objects with significant mAP gains on MS-COCO and Pascal VOC.

ABSTRACT

Conventional object detection models require large amounts of training data. In comparison, humans can recognize previously unseen objects by merely knowing their semantic description. To mimic similar behaviour, zero-shot object detection aims to recognize and localize 'unseen' object instances by using only their semantic information. The model is first trained to learn the relationships between visual and semantic domains for seen objects, later transferring the acquired knowledge to totally unseen objects. This setting gives rise to the need for correct alignment between visual and semantic concepts, so that the unseen objects can be identified using only their semantic attributes. In this paper, we propose a novel loss function called 'Polarity loss', that promotes correct visual-semantic alignment for an improved zero-shot object detection. On one hand, it refines the noisy semantic embeddings via metric learning on a 'Semantic vocabulary' of related concepts to establish a better synergy between visual and semantic domains. On the other hand, it explicitly maximizes the gap between positive and negative predictions to achieve better discrimination between seen, unseen and background objects. Our approach is inspired by embodiment theories in cognitive science, that claim human semantic understanding to be grounded in past experiences (seen objects), related linguistic concepts (word vocabulary) and visual perception (seen/unseen object images). We conduct extensive evaluations on MS-COCO and Pascal VOC datasets, showing significant improvements over state of the art.

Motivation & Objective

• Motivate zero-shot object detection (ZSD) to locate and recognize unseen objects using semantic descriptions.
• Improve visual-semantic alignment by refining semantic embeddings and maximizing margins between positive and negative predictions.
• Introduce Polarity Loss that combines a focal-style loss with a margin-based penalty to handle object-background imbalance and unseen-background confusion.
• Leverage an external vocabulary and metric learning to reshape semantic representations for better alignment with visual features.
• Evaluate on MS-COCO and Pascal VOC with new seen/unseen splits and report improvements over prior methods.

Proposed method

• Propose Polarity Loss, which builds on Focal Loss by adding a monotonic penalty to maximize the margin between positive and negative class predictions (Eq. 6).
• Introduce a penalty function f_p, implemented as a beta-weighted sigmoid (or softplus variant), to encourage p^i < p^ℓ for negatives relative to the ground-truth class ℓ (Eq. 5).
• Extend to an end-to-end single-stage detector (RetinaNet) with classification and regression heads that project features into semantically meaningful word-vector space (Eq. 9/10).
• Embed semantic information via word-vectors with either fixed seen-word embeddings or a vocabulary-based metric learning mechanism (δ(W_s MD)) to relate visual features to a broader vocabulary (Fig. 5c).
• Apply a vocabulary metric learning approach to update semantic representations, using a dictionary D and learnable attention M to obtain p_v and integrate semantics into both classification and regression branches (Eq. 10).
• Provide two implementation variants for semantics in the classifier: (a) fixed W_s with a simple projection, and (b) vocabulary-attention variant with M and D (Fig. 7).
• Train with the Polarity Loss for classification and standard Smooth L1 for regression; use anchor-based sampling and NMS for inference; compute unseen scores with a separate unseen-weighted projection (Eq. 11).

Experimental results

Research questions

• RQ1 How can visual-semantic alignment be enhanced for zero-shot object detection to distinguish unseen objects from background?
• RQ2 Can a margin-based polarity loss improve discrimination between seen, unseen, and background classes beyond standard focal loss?
• RQ3 Does incorporating an external vocabulary and vocabulary-driven metric learning improve zero-shot detection performance?
• RQ4 How does a single-stage detector like RetinaNet perform for ZSD with the proposed loss compared to prior ZSD methods?
• RQ5 What is the impact of new MS-COCO seen/unseen splits on zero-shot detection outcomes?

Key findings

• Significant improvements over prior work on MS-COCO and Pascal VOC for zero-shot detection.
• Reported absolute gains of 9.3 and 7.6 in mAP over two prior methods on MS-COCO (as cited) and related results on others.
• The approach achieves best mAP on unseen classes in their experiments.
• Demonstrates that vocabulary-driven semantic refinement reduces unseen-background confusion and improves cross-domain alignment.
• Shows that a single-stage detector with Polarity Loss can effectively detect and localize unseen objects while maintaining performance on seen classes.