[Paper Review] ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval

Mengjun Cheng, Y. J. Sun+9 more|arXiv (Cornell University)|Mar 31, 2022

Multimodal Machine Learning Applications4 citations

TL;DR

ViSTA proposes a full transformer-based framework that unifies cross-modal retrieval for both scene text-aware and scene text-free scenarios by fusing visual and scene text features through a novel fusion token mechanism and dual contrastive learning. It achieves state-of-the-art performance, improving Recall@1 by 8.4% on scene text-aware retrieval and running at least three times faster than prior methods.

ABSTRACT

Visual appearance is considered to be the most important cue to understand images for cross-modal retrieval, while sometimes the scene text appearing in images can provide valuable information to understand the visual semantics. Most of existing cross-modal retrieval approaches ignore the usage of scene text information and directly adding this information may lead to performance degradation in scene text free scenarios. To address this issue, we propose a full transformer architecture to unify these cross-modal retrieval scenarios in a single $ extbf{Vi}$sion and $ extbf{S}$cene $ extbf{T}$ext $ extbf{A}$ggregation framework (ViSTA). Specifically, ViSTA utilizes transformer blocks to directly encode image patches and fuse scene text embedding to learn an aggregated visual representation for cross-modal retrieval. To tackle the modality missing problem of scene text, we propose a novel fusion token based transformer aggregation approach to exchange the necessary scene text information only through the fusion token and concentrate on the most important features in each modality. To further strengthen the visual modality, we develop dual contrastive learning losses to embed both image-text pairs and fusion-text pairs into a common cross-modal space. Compared to existing methods, ViSTA enables to aggregate relevant scene text semantics with visual appearance, and hence improve results under both scene text free and scene text aware scenarios. Experimental results show that ViSTA outperforms other methods by at least $\bf{8.4}\%$ at Recall@1 for scene text aware retrieval task. Compared with state-of-the-art scene text free retrieval methods, ViSTA can achieve better accuracy on Flicker30K and MSCOCO while running at least three times faster during the inference stage, which validates the effectiveness of the proposed framework.

Motivation & Objective

• To address the limitation of existing cross-modal retrieval models that degrade in performance when scene text is absent or noisy.
• To unify scene text-aware and scene text-free cross-modal retrieval into a single, scalable framework without sacrificing performance in either scenario.
• To enhance visual representation by effectively aggregating relevant scene text semantics only when present, using a lightweight fusion mechanism.
• To strengthen visual modality robustness through dual contrastive learning that preserves alignment even when scene text is missing.
• To enable efficient, high-speed inference by leveraging a dual-encoder-style transformer architecture with minimal computational overhead.

Proposed method

• ViSTA employs a full transformer architecture to encode image patches and scene text embeddings separately, then fuses them via a dedicated fusion token that exchanges modality-specific information.
• The fusion token mechanism enables selective, attention-based interaction between visual and scene text features, concentrating on the most salient features in each modality.
• Dual contrastive learning is introduced: one loss aligns image-text pairs, and another aligns image-fusion pairs, improving visual feature robustness under modality absence.
• The model uses a dual-encoder setup for efficient inference, pre-computing image and text features offline, enabling fast similarity computation at scale.
• The fusion mechanism is applied across multiple layers (up to 4), with ablation studies showing optimal performance at 4 layers.
• Ablation experiments compare fusion strategies, including global attention, cross-attention, and late fusion, demonstrating the superiority of the fusion token approach.

Experimental results

Research questions

• RQ1 Can a unified cross-modal retrieval framework effectively handle both scene text-aware and scene text-free scenarios without performance degradation?
• RQ2 How can scene text information be selectively aggregated with visual features to avoid interference when scene text is absent?
• RQ3 What is the optimal fusion mechanism for integrating visual and scene text features in a transformer-based architecture?
• RQ4 Does dual contrastive learning improve visual representation robustness when scene text is missing or noisy?
• RQ5 Can the proposed method achieve high accuracy while maintaining fast inference speed for large-scale retrieval?

Key findings

• ViSTA achieves a 52.5% Recall@1 on the CTC-1K benchmark for image-to-text retrieval in scene text-aware scenarios, outperforming prior state-of-the-art methods by at least 8.4%.
• The model maintains strong performance in scene text-free scenarios, achieving 47.0% Recall@1 on CTC-1K, comparable to or better than SOTA methods.
• ViSTA runs at least three times faster than existing methods during inference, due to its dual-encoder design and efficient fusion mechanism.
• Ablation studies confirm that the fusion token strategy outperforms global attention, cross-attention, and late fusion, achieving 52.5% R@1 on image-to-text retrieval.
• Dual contrastive learning significantly improves performance, with the full loss setting achieving 52.5% R@1, compared to 46.6% when only one contrastive loss is used.
• Increasing the number of fusion layers improves performance, peaking at 4 layers with 52.5% R@1, indicating deeper fusion enhances feature learning.