[Paper Review] Cross-Modal Scene Networks

Yusuf Aytar, Lluís Castrejón+3 more|arXiv (Cornell University)|Oct 27, 2016

Advanced Image and Video Retrieval Techniques43 references4 citations

TL;DR

This paper proposes a method to learn cross-modal scene representations that are aligned across diverse modalities—such as natural images, sketches, text descriptions, and clip art—without requiring paired data. By regularizing convolutional neural networks with modality-invariant contrastive loss and statistical regularization, the model learns a shared representation where units activate on consistent scene concepts regardless of modality, enabling effective zero-shot retrieval and reconstruction across modalities.

ABSTRACT

People can recognize scenes across many different modalities beyond natural images. In this paper, we investigate how to learn cross-modal scene representations that transfer across modalities. To study this problem, we introduce a new cross-modal scene dataset. While convolutional neural networks can categorize scenes well, they also learn an intermediate representation not aligned across modalities, which is undesirable for cross-modal transfer applications. We present methods to regularize cross-modal convolutional neural networks so that they have a shared representation that is agnostic of the modality. Our experiments suggest that our scene representation can help transfer representations across modalities for retrieval. Moreover, our visualizations suggest that units emerge in the shared representation that tend to activate on consistent concepts independently of the modality.

Motivation & Objective

• To enable cross-modal scene understanding across highly dissimilar modalities (e.g., images, text, sketches) without paired training examples.
• To learn a shared, modality-agnostic representation that preserves high-level scene concepts across modalities.
• To address the challenge of domain shift in cross-modal transfer by enforcing alignment in intermediate feature spaces.
• To demonstrate that strong alignment can emerge from weak supervision (only scene category labels) without explicit correspondence annotations.

Proposed method

• Introduce a new cross-modal scene dataset with 500+ scene categories across five modalities: natural images, clip art, line drawings, spatial text, and descriptive text.
• Train a shared Siamese-like network with modality-specific encoders and a shared representation head to learn joint features.
• Apply modality-invariant contrastive loss to encourage identical feature representations for the same scene across different modalities.
• Use statistical regularization (e.g., batch normalization and feature whitening) to further reduce modality-specific biases in the shared features.
• Train an inverting network on the shared pool5 features to reconstruct input images from any modality, enabling visualization of preserved concepts.
• Leverage t-SNE visualization and qualitative reconstruction analysis to evaluate modality invariance and feature alignment.

Experimental results

Research questions

• RQ1 Can a deep neural network learn a shared, modality-agnostic representation for scenes using only category-level supervision and no paired data?
• RQ2 Do units in the shared representation automatically learn to detect consistent high-level concepts (e.g., 'bed', 'car') across different modalities?
• RQ3 To what extent can the learned representation enable zero-shot cross-modal retrieval when training data for a modality is missing?
• RQ4 How well can the shared representation preserve scene-level statistics to enable reconstruction of images from other modalities?
• RQ5 Does the model’s representation become invariant to modality, discarding modality-specific details while retaining semantic content?

Key findings

• The proposed method outperforms fine-tuning baselines in cross-modal retrieval, achieving higher mAP on zero-shot retrieval benchmarks.
• The model enables effective zero-shot retrieval even when training data for a specific modality is absent, demonstrating robust generalization.
• t-SNE visualizations show that the full method reduces modality-specific clustering, indicating improved modality invariance compared to baselines.
• Reconstruction results demonstrate that the shared representation preserves key scene concepts and spatial layout across modalities, despite information compression.
• Units in the shared representation activate on consistent scene concepts (e.g., beds, cars) regardless of input modality, suggesting emergent disentanglement of semantic content.
• The method successfully reconstructs images from diverse modalities using only the shared features, with reconstructions preserving scene composition and major objects.