[Paper Review] ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration

Yuhao Cui, Yu Zhou+5 more|arXiv (Cornell University)|Aug 16, 2021

Multimodal Machine Learning Applications52 references4 citations

TL;DR

ROSITA proposes a vision-and-language pretraining method that enhances fine-grained semantic alignments by integrating cross-modal and intra-modal knowledge into a unified scene graph, using a novel structural knowledge masking (SKM) strategy to improve masked language and region modeling. It achieves state-of-the-art performance across six benchmark datasets on three vision-and-language tasks, outperforming existing methods through more robust and precise cross-modal alignment.

ABSTRACT

Vision-and-language pretraining (VLP) aims to learn generic multimodal representations from massive image-text pairs. While various successful attempts have been proposed, learning fine-grained semantic alignments between image-text pairs plays a key role in their approaches. Nevertheless, most existing VLP approaches have not fully utilized the intrinsic knowledge within the image-text pairs, which limits the effectiveness of the learned alignments and further restricts the performance of their models. To this end, we introduce a new VLP method called ROSITA, which integrates the cross- and intra-modal knowledge in a unified scene graph to enhance the semantic alignments. Specifically, we introduce a novel structural knowledge masking (SKM) strategy to use the scene graph structure as a priori to perform masked language (region) modeling, which enhances the semantic alignments by eliminating the interference information within and across modalities. Extensive ablation studies and comprehensive analysis verifies the effectiveness of ROSITA in semantic alignments. Pretrained with both in-domain and out-of-domain datasets, ROSITA significantly outperforms existing state-of-the-art VLP methods on three typical vision-and-language tasks over six benchmark datasets.

Motivation & Objective

• To address the limitation of existing vision-and-language pretraining (VLP) methods in fully utilizing intrinsic knowledge from image-text pairs.
• To improve fine-grained semantic alignments between image regions and text words, which are critical for downstream V+L tasks.
• To integrate both cross-modal knowledge (between image and text) and intra-modal knowledge (within each modality) in a unified framework.
• To develop a robust masking strategy that leverages structured knowledge for more effective pretraining.
• To demonstrate consistent performance gains across diverse vision-and-language benchmarks.

Proposed method

• Constructs a unified scene graph centered on anchor objects, encoding cross-modal relationships (e.g., 'grass' ↔ 'steppe') and intra-modal relationships (e.g., spatially related regions or contextually related words).
• Introduces a novel Structural Knowledge Masking (SKM) strategy that masks anchor objects and selectively masks related knowledge entries based on their structural roles in the scene graph.
• Uses independent masking probabilities per knowledge entry to allow fine-grained control over information masking, improving robustness compared to uniform probability strategies.
• Integrates SKM seamlessly into standard masked language (region) modeling objectives used in existing VLP models.
• Pretrains the model on both in-domain and out-of-domain datasets using the unified scene graph and SKM to enhance cross-modal alignment.

Figure 1 . Schematic of the knowledge integration strategies of three VLP methods, i.e. , OSCAR (Li et al . , 2020b ) , ERNIE-ViL (Yu et al . , 2021 ) , and our ROSITA. OSCAR and ERNIE-ViL only exploit the intra-modal knowledge from the image and text modalities, respectively. In contrast, ROSITA si

Experimental results

Research questions

• RQ1 Can integrating both cross-modal and intra-modal knowledge in a unified scene graph improve vision-and-language semantic alignment?
• RQ2 How does the proposed structural knowledge masking (SKM) strategy compare to standard masking methods in enhancing alignment learning?
• RQ3 Does using independent masking probabilities per knowledge entry lead to better generalization than fixed, identical probabilities?
• RQ4 Can the unified knowledge integration strategy consistently improve performance across diverse vision-and-language benchmarks?
• RQ5 To what extent does ROSITA improve attention-based cross-modal alignment compared to baseline models like UNITER?

Key findings

• ROSITA significantly outperforms existing state-of-the-art VLP methods on six benchmark datasets across three core vision-and-language tasks: visual question answering, image-text retrieval, and referring expression comprehension.
• The full ROSITA model consistently surpasses all ablated variants, confirming the effectiveness of jointly integrating cross- and intra-modal knowledge.
• The SKM strategy with independent masking probabilities achieves superior performance compared to identical probability strategies, which are sensitive to hyperparameter choices.
• Visualized attention maps show that ROSITA learns precise cross-modal alignments—e.g., correctly aligning a masked 'ramp' region to the word 'ramp'—while baseline models like UNITER fail and attend to incorrect tokens such as 'skate'.
• When multiple regions or words are masked simultaneously, ROSITA maintains accurate alignment, correctly linking 'bowl & carrots' to 'bowl' and 'carrots', and 'man', 'candles', and 'cake' to their respective image regions.

Figure 2 . The flowchart of knowledge extraction given an image-text pair. It consists of two main stages, namely the unified scene graph construction and knowledge representation.