[Paper Review] See, Hear, and Read: Deep Aligned Representations

Yusuf Aytar, Carl Vondrick+1 more|arXiv (Cornell University)|Jun 3, 2017

Multimodal Machine Learning Applications41 references68 citations

TL;DR

The paper trains a deep cross-modal network to learn a shared, aligned representation across vision, sound, and language using large-scale synchronized data, enabling cross-modal retrieval and transfer without requiring image-sound-text pairs during training.

ABSTRACT

We capitalize on large amounts of readily-available, synchronous data to learn a deep discriminative representations shared across three major natural modalities: vision, sound and language. By leveraging over a year of sound from video and millions of sentences paired with images, we jointly train a deep convolutional network for aligned representation learning. Our experiments suggest that this representation is useful for several tasks, such as cross-modal retrieval or transferring classifiers between modalities. Moreover, although our network is only trained with image+text and image+sound pairs, it can transfer between text and sound as well, a transfer the network never observed during training. Visualizations of our representation reveal many hidden units which automatically emerge to detect concepts, independent of the modality.

Motivation & Objective

• Capitalize on large-scale synchronized data to learn representations shared across vision, sound, and language.
• Develop a deep network that produces a common representation for images, sounds, and sentences.
• Demonstrate cross-modal retrieval, classification transfer, and emergent modality-independent concepts.
• Show that alignment can transfer between modalities (e.g., text and sound) even without direct training pairs.

Proposed method

• Propose a three-branch cross-modal convolutional network with modality-specific early layers and shared upper layers.
• Use two alignment losses: a model transfer loss where a student modality predicts class probabilities from a teacher image model, with shared upper weights; and a ranking loss enforcing closeness of matched pairs and separation of mismatched ones in the shared space.
• Represent inputs as spectrograms (sound), word2vec-embedded sentences (text), and images, processed by respective CNNs and fed into a 1000-dim softmax shared representation.
• Train with synchronized image+sound and image+text pairs, using imagery as a bridge to enable text-sound alignment; evaluate on cross-modal retrieval and classifier transfer.
• Visualize hidden units to show emergent, modality-agnostic concept detectors.

Experimental results

Research questions

• RQ1 Can an aligned, discriminative representation be learned across vision, sound, and language from large-scale synchronized data?
• RQ2 Does the learned representation enable effective cross-modal retrieval beyond image-text, including image-sound and sound-text transfers?
• RQ3 Can classifiers trained in one modality transfer to others using the aligned representation without target-modality labels?
• RQ4 Do internal units emerge that detect concepts independent of modality?

Key findings

• Cross-modal retrieval significantly outperforms baselines on vision-sound-text tasks, with improved alignment over linear regression and CCA on several modality pairs.
• Sound-text retrieval via the shared representation substantially outperforms baselines like cluster CCA and linear regression by large margins.
• Classifier transfer across modalities achieves up to 10% accuracy gains over baselines in cross-modal settings.
• The representation enables transfer from vision and text to sound and vice versa, even though sound-text pairs were not present during training.
• Hidden units in the shared layers automatically detect high-level concepts and often respond to objects independent of modality.