[Paper Review] Order-Embeddings of Images and Language

Ivan Vendrov, Ryan Kiros+2 more|arXiv (Cornell University)|Nov 19, 2015

Multimodal Machine Learning Applications28 references87 citations

TL;DR

This paper introduces order-embeddings, a method that learns distributed representations preserving the partial order structure of the visual-semantic hierarchy—where images, sentences, and words relate via abstraction (e.g., 'dog' → 'mammal' or 'woman walking her dog' → 'person walking'). By replacing distance-based similarity with order-preserving constraints in a learned embedding space, the approach improves performance on hypernym prediction and image-caption retrieval, outperforming prior state-of-the-art methods on these tasks.

ABSTRACT

Hypernymy, textual entailment, and image captioning can be seen as special cases of a single visual-semantic hierarchy over words, sentences, and images. In this paper we advocate for explicitly modeling the partial order structure of this hierarchy. Towards this goal, we introduce a general method for learning ordered representations, and show how it can be applied to a variety of tasks involving images and language. We show that the resulting representations improve performance over current approaches for hypernym prediction and image-caption retrieval.

Motivation & Objective

• Address the limitation of existing embedding methods that rely on distance preservation, which fails to capture the antisymmetric and transitive nature of semantic hierarchies.
• Model the visual-semantic hierarchy as a partial order, where concepts like images, captions, and words are related via abstraction (e.g., 'dog' is a hypernym of 'poodle').
• Develop a general framework to learn embeddings that preserve the partial order structure, enabling better performance on tasks involving hierarchical relationships.
• Demonstrate the effectiveness of order-embeddings across three tasks: hypernym prediction, caption-image retrieval, and textual entailment.
• Provide a unified approach that integrates with existing relational learning models by replacing their similarity comparison with an order-preserving operation.

Proposed method

• Define the problem as partial order completion: predict whether an unseen pair (u,v) is ordered based on learned embeddings.
• Propose order-embeddings as a mapping f: (X, ⪯_X) → (Y, ⪯_Y) such that u ⪯_X v if and only if f(u) ⪯_Y f(v), ensuring exact order preservation.
• Use the reversed product order on ℝ₊^N as the embedding space order, which supports both abstraction (meet) and composition (join) operations.
• Train with a margin-based loss that penalizes order violations: if f(u) ≼ f(v) but u is not ≤ v, the model incurs a penalty.
• Apply a GRU-based encoder to embed sentences and images into the shared space, with unit L2-normalized embeddings.
• Use a contrastive learning setup with hard negative mining, where positive pairs are ordered and negative pairs are not.

Experimental results

Research questions

• RQ1 Can order-preserving embeddings improve performance on hypernym prediction compared to distance-based similarity methods?
• RQ2 Can order-embeddings enhance image-caption retrieval by modeling the hierarchical relationship between captions and images?
• RQ3 Can the same framework effectively model textual entailment, where one sentence logically implies another?
• RQ4 How does enforcing partial order structure in embeddings compare to symmetric similarity in capturing semantic hierarchy?
• RQ5 To what extent can order-embeddings unify disparate NLP and vision tasks under a single hierarchical representation framework?

Key findings

• Order-embeddings achieve 88.6% accuracy on the 2-class SNLI entailment task, outperforming the skip-thought baseline (87.7%) and the EOP classifier (75.0%).
• On hypernym prediction, order-embeddings significantly outperform previous state-of-the-art methods by explicitly modeling the antisymmetric and transitive nature of hypernymy.
• For image-caption retrieval, order-embeddings improve performance over distance-based baselines by preserving the hierarchical abstraction from image to caption.
• The method achieves near-state-of-the-art performance on textual entailment while being simpler than attention-based models, suggesting strong inductive bias from order constraints.
• Visualizations in the supplementary material reveal novel vector regularities, such as consistent geometric patterns reflecting hierarchical abstraction.
• The approach generalizes well across tasks, demonstrating that order-embeddings can unify modeling of hypernymy, textual entailment, and image-caption relationships in a single framework.