[Paper Review] Compensating Supervision Incompleteness with Prior Knowledge in Semantic Image Interpretation

Ivan Donadello, Luciano Serafini|arXiv (Cornell University)|Oct 1, 2019

Multimodal Machine Learning Applications31 references4 citations

TL;DR

This paper proposes using Logic Tensor Networks (LTNs) to address zero-shot visual relationship detection by integrating logical background knowledge with supervised learning, significantly improving performance on the Visual Relationship Dataset. The method outperforms state-of-the-art approaches by leveraging logical constraints to compensate for incomplete supervision, achieving 77.16% recall at 100 for predicate detection.

ABSTRACT

Semantic Image Interpretation is the task of extracting a structured semantic description from images. This requires the detection of visual relationships: triples (subject,relation,object) describing a semantic relation between a subject and an object. A pure supervised approach to visual relationship detection requires a complete and balanced training set for all the possible combinations of (subject, relation, object). However, such training sets are not available and would require a prohibitive human effort. This implies the ability of predicting triples which do not appear in the training set. This problem is called zero-shot learning. State-of-the-art approaches to zero-shot learning exploit similarities among relationships in the training set or external linguistic knowledge. In this paper, we perform zero-shot learning by using Logic Tensor Networks, a novel Statistical Relational Learning framework that exploits both the similarities with other seen relationships and background knowledge, expressed with logical constraints between subjects, relations and objects. The experiments on the Visual Relationship Dataset show that the use of logical constraints outperforms the current methods. This implies that background knowledge can be used to alleviate the incompleteness of training sets.

Motivation & Objective

• Address the challenge of zero-shot visual relationship detection due to incomplete and imbalanced training data in semantic image interpretation.
• Overcome the limitations of purely supervised or linguistic knowledge-based methods by incorporating explicit logical background knowledge.
• Improve generalization to unseen visual relationships by combining data-driven learning with logical reasoning.
• Demonstrate that logical constraints can effectively reduce the impact of annotation incompleteness in real-world datasets.
• Evaluate the effectiveness of novel features and a new loss function in enhancing model performance within the LTN framework.

Proposed method

• Employ Logic Tensor Networks (LTNs), a Statistical Relational Learning framework, to jointly learn from visual data and logical constraints.
• Introduce geometric features capturing spatial relationships between bounding boxes to improve triple prediction accuracy.
• Formulate background knowledge as first-order logic axioms, such as ∀x,y(ride(x,y)→elephant(y)∨horse(y)) and ¬ride(x,y) if x is not a rider.
• Use a novel loss function based on the harmonic mean of truth values of logical axioms to stabilize training and improve constraint adherence.
• Train the model end-to-end using a differentiable logic framework that allows backpropagation through logical formulas.
• Apply t-norm-based logic operations to compute truth values of logical statements within the neural network optimization process.

Experimental results

Research questions

• RQ1 Can logical background knowledge effectively compensate for supervision incompleteness in visual relationship detection?
• RQ2 How does the integration of logical constraints improve zero-shot generalization compared to data-only or linguistic-only methods?
• RQ3 What is the impact of geometric features and a new loss function on LTN performance in visual relationship detection?
• RQ4 How do logical constraints reduce false positive predictions for semantically incoherent relationships (e.g., animals riding humans)?
• RQ5 To what extent does the proposed method scale and generalize across different types of visual relationships?

Key findings

• The model with logical constraints (T_prior) achieves 77.16% recall at 100 for predicate detection, outperforming the previous state-of-the-art method (LKD) at 74.65%.
• The inclusion of geometric features improves performance across all tasks, with a 2.45% absolute gain in predicate detection recall at 100 compared to the baseline.
• The new harmonic mean-based loss function enhances constraint adherence and leads to a 2.45% improvement in predicate detection over the standard t-norm loss.
• Logical constraints effectively suppress semantically incoherent predictions, such as animals riding humans, by encoding domain-specific restrictions.
• The LTN framework reduces the search space from O(|P1|²|P2|) to O(|P1| + |P2|), significantly improving scalability and reducing parameter count.
• Ablation studies confirm that both the new features and the new loss function are essential for optimal performance, with each contributing independently to the gains.