[Paper Review] Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation

Ruichi Yu, Ang Li+2 more|arXiv (Cornell University)|Jul 28, 2017

Multimodal Machine Learning ApplicationsComputer Science28 references57 citations

TL;DR

The paper presents a teacher-student framework that distills internal and external linguistic knowledge into a visual relationship detector, improving predicate prediction, especially in zero-shot cases.

ABSTRACT

Understanding visual relationships involves identifying the subject, the object, and a predicate relating them. We leverage the strong correlations between the predicate and the (subj,obj) pair (both semantically and spatially) to predict the predicates conditioned on the subjects and the objects. Modeling the three entities jointly more accurately reflects their relationships, but complicates learning since the semantic space of visual relationships is huge and the training data is limited, especially for the long-tail relationships that have few instances. To overcome this, we use knowledge of linguistic statistics to regularize visual model learning. We obtain linguistic knowledge by mining from both training annotations (internal knowledge) and publicly available text, e.g., Wikipedia (external knowledge), computing the conditional probability distribution of a predicate given a (subj,obj) pair. Then, we distill the knowledge into a deep model to achieve better generalization. Our experimental results on the Visual Relationship Detection (VRD) and Visual Genome datasets suggest that with this linguistic knowledge distillation, our model outperforms the state-of-the-art methods significantly, especially when predicting unseen relationships (e.g., recall improved from 8.45% to 19.17% on VRD zero-shot testing set).

Motivation & Objective

• Capture and predict visual relationships as triples ⟨subject, predicate, object⟩ by jointly modeling the three components.
• Regularize a deep visual model with linguistic knowledge to address long-tail and unseen relationships.
• Leverage both internal (training annotations) and external (public text) linguistic statistics via knowledge distillation.

Proposed method

• Model predicates jointly with subject and object representations and their spatial configuration.
• Construct a teacher network using linguistic knowledge P(pred|subj,obj) and distill it into a student network during training.
• Collect linguistic knowledge from training annotations and Wikipedia, combining them to form the teacher's guidance.
• Use semantic embeddings for subjects/objects and spatial features to condition predicate probabilities.
• Train end-to-end with a loss that blends ground-truth supervision and teacher guidance (KL-like distillation).
• Evaluate using Recall@k on VRD and Visual Genome datasets, including zero-shot splits.

Experimental results

Research questions

• RQ1 Can linguistic statistics (internal and external) regularize deep visual relationship models to improve generalization?
• RQ2 How does combining teacher and student networks affect performance in seen and zero-shot scenarios?
• RQ3 What is the impact of semantic and spatial representations on predicate prediction accuracy?
• RQ4 Do external knowledge sources (e.g., Wikipedia) help or hurt when integrated with internal training data?

Key findings

• Linguistic knowledge distillation significantly improves predicate prediction compared to a purely data-driven baseline.
• Zero-shot recall on VRD improves from 8.45% to 19.17% with LK distillation.
• Combining teacher and student predictions (T+S) yields best results, outperforming baselines across seen and zero-shot settings.
• Using subject/object semantic representations and spatial features enhances predictive power and generalization.
• External knowledge alone can be noisy, but when combined with internal knowledge and visual data, LK distillation remains beneficial.