[Paper Review] UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations

Hao Wu, Jiayuan Mao+5 more|arXiv (Cornell University)|Apr 11, 2019

Multimodal Machine Learning Applications45 references4 citations

TL;DR

UniVSE proposes a unified visual-semantic embedding space that factorizes textual and visual concepts into objects, attributes, relations, and scenes, enabling fine-grained cross-modal alignment through contrastive learning on semantic components and enforcing semantic coverage to improve robustness. The method achieves state-of-the-art performance on cross-modal retrieval and effectively defends against text-domain adversarial attacks while enabling visual-guided semantic parsing.

ABSTRACT

We propose Unified Visual-Semantic Embeddings (UniVSE) for learning a joint space of visual and textual concepts. The space unifies the concepts at different levels, including objects, attributes, relations, and full scenes. A contrastive learning approach is proposed for the fine-grained alignment from only image-caption pairs. Moreover, we present an effective approach for enforcing the coverage of semantic components that appear in the sentence. We demonstrate the robustness of Unified VSE in defending text-domain adversarial attacks on cross-modal retrieval tasks. Such robustness also empowers the use of visual cues to resolve word dependencies in novel sentences.

Motivation & Objective

• To establish a unified, factorized visual-semantic embedding space that aligns objects, attributes, relations, and full scenes across vision and language.
• To resolve referential ambiguity in visual-textual alignment using cross-situational learning with contrastive image-caption pairs.
• To mitigate dataset biases—such as object co-occurrence—by enforcing semantic coverage in caption encoding.
• To improve robustness against text-domain adversarial attacks in cross-modal retrieval tasks.
• To enable visual cues to assist in parsing novel sentences, particularly in resolving word dependencies and semantic roles.

Proposed method

• The model uses contrastive learning on semantic components (e.g., nouns, prepositional phrases) rather than full sentences or images, enabling fine-grained alignment between textual phrases and visual regions.
• A novel contrastive example mining strategy identifies pairs of images and captions that differ in only one semantic component (e.g., 'wall' vs. 'shelf'), resolving referential ambiguity.
• A semantic coverage regularization is applied to the caption encoder, ensuring that every semantic component in the sentence contributes meaningfully to the global caption embedding.
• The unified embedding space jointly represents visual regions and textual phrases at multiple levels: objects (noun phrases), attributes (prenominal phrases), relations (verbs/prepositions), and full scenes (sentences).
• The model computes similarity maps between image regions and query embeddings to visualize and verify alignment of semantic components with corresponding visual regions.
• For semantic parsing, the model uses the unified embedding space to score possible dependency combinations (e.g., subject-object of a verb) and selects the highest-scoring one based on visual matching.

Experimental results

Research questions

• RQ1 Can contrastive learning on semantic components improve fine-grained alignment between vision and language?
• RQ2 Does enforcing semantic coverage in caption encoding enhance robustness to adversarial perturbations in the text domain?
• RQ3 Can visual cues from the unified embedding space improve the accuracy of semantic parsing in ambiguous sentences?
• RQ4 How does the unified factorized representation compare to end-to-end sentence-level alignment in cross-modal retrieval tasks?
• RQ5 To what extent can visual information resolve syntactic ambiguities in novel sentences not seen during training?

Key findings

• UniVSE achieves state-of-the-art performance on cross-modal retrieval across all query types, including single words, noun phrases, relational phrases, and full sentences, outperforming VSE++, VSE-C, and random baselines.
• The model trained with all types of contrastive components (UniVSE with all) achieves the highest retrieval accuracy, with a significant margin over baselines, especially on fine-grained queries like single nouns.
• Semantic coverage regularization significantly improves robustness, enabling UniVSE to defend against text-domain adversarial attacks that fool standard models.
• In semantic parsing with visual cues, UniVSE achieves 64.82% accuracy on attributed object dependency recovery and 62.69% on relational phrase parsing, substantially outperforming VSE++ (41.12% and 43.31%) and VSE-C (43.44% and 41.08%).
• Visualization of similarity maps confirms that UniVSE successfully aligns specific semantic components (e.g., 'clock', 'wall') with corresponding image regions, demonstrating precise spatial grounding.