[Paper Review] Classification by Attention: Scene Graph Classification with Prior Knowledge

Sahand Sharifzadeh, Sina Moayed Baharlou+1 more|arXiv (Cornell University)|Nov 19, 2020

Advanced Graph Neural Networks4 citations

TL;DR

This paper proposes a multi-task learning framework that integrates visual perception and commonsense prior knowledge via an attention-based classification layer, enabling iterative reasoning over object relations. By treating classification as an attention mechanism, the model learns to inject prior knowledge into visual representations, achieving 26% higher scene graph classification accuracy and 36% improvement in predicate prediction with only 1% of annotated data when combined with self-supervised pretraining.

ABSTRACT

A major challenge in scene graph classification is that the appearance of objects and relations can be significantly different from one image to another. Previous works have addressed this by relational reasoning over all objects in an image or incorporating prior knowledge into classification. Unlike previous works, we do not consider separate models for perception and prior knowledge. Instead, we take a multi-task learning approach, where we implement the classification as an attention layer. This allows for the prior knowledge to emerge and propagate within the perception model. By enforcing the model also to represent the prior, we achieve a strong inductive bias. We show that our model can accurately generate commonsense knowledge and that the iterative injection of this knowledge to scene representations leads to significantly higher classification performance. Additionally, our model can be fine-tuned on external knowledge given as triples. When combined with self-supervised learning and with 1% of annotated images only, this gives more than 3% improvement in object classification, 26% in scene graph classification, and 36% in predicate prediction accuracy.

Motivation & Objective

• To address the challenge of visual variation in object and relation appearances across images, which hampers scene graph classification.
• To overcome the limitations of treating perception and prior knowledge as separate, independent components in existing models.
• To enable mutual knowledge transfer between visual features and relational commonsense knowledge through shared, end-to-end training.
• To demonstrate that injecting prior knowledge via attention improves generalization, especially under low-data regimes.
• To show that the model can be fine-tuned on external knowledge triples and still achieve high performance with minimal supervision.

Proposed method

• Reformulate classification as an attention layer instead of a fully connected layer to allow for the propagation of prior knowledge within the perception model.
• Use multi-task learning to jointly train visual representation learning and prior knowledge integration, sharing parameters across both tasks.
• Inject prior knowledge from a knowledge graph (via embeddings) into image-based representations through iterative assimilation.
• Leverage self-supervised pretraining (e.g., BYOL) to initialize the model, then fine-tune on visual genome with limited annotations.
• Represent schemata as shared embeddings that encode both image-based and relational knowledge, enabling generalization beyond seen triples.
• Perform link prediction on schema representations to evaluate the model’s understanding of relational commonsense without visual input.

Experimental results

Research questions

• RQ1 Can attention-based classification enable the integration of prior knowledge into visual representations without separate modeling?
• RQ2 Does iterative injection of prior knowledge improve scene graph classification under low-data settings?
• RQ3 Can the model learn and generalize commonsense relational knowledge (e.g., 'Fruits are in Bowls') even when visual cues are weak?
• RQ4 How effective is the model when fine-tuned on external knowledge triples, especially with minimal supervision?
• RQ5 To what extent does the model’s internal representation reflect meaningful semantic and relational structure?

Key findings

• With only 1% of annotated images, the model achieved a 26% improvement in scene graph classification accuracy compared to baseline models.
• Predicate prediction accuracy improved by 36% under the same low-data regime, demonstrating strong generalization to unseen relations.
• The model achieved near-full performance in predicate prediction when using pre-trained schemata and external knowledge triples, even without visual input.
• t-SNE visualizations confirmed that object representations became more separable after injecting prior knowledge, indicating improved semantic clustering.
• Link prediction on schema representations alone achieved non-trivial accuracy, showing that the model internalized relational commonsense knowledge.
• The model outperformed baselines across all metrics, especially in low-data regimes, and demonstrated robustness through statistical stability over multiple random seeds.