[Paper Review] Learning to Predict: A Fast Re-constructive Method to Generate Multimodal Embeddings

Guillem Collell, Teddy Zhang+1 more|arXiv (Cornell University)|Mar 25, 2017

Topic Modeling21 references3 citations

TL;DR

This paper proposes a fast, cognitively inspired method to generate multimodal embeddings by learning a language-to-vision mapping that reconstructs visual representations from text, creating associative, re-constructive multimodal embeddings. The approach outperforms strong unimodal and state-of-the-art multimodal baselines on seven concept similarity benchmarks, especially in zero-shot settings, demonstrating more 'human-like' semantic judgments.

ABSTRACT

Integrating visual and linguistic information into a single multimodal representation is an unsolved problem with wide-reaching applications to both natural language processing and computer vision. In this paper, we present a simple method to build multimodal representations by learning a language-to-vision mapping and using its output to build multimodal embeddings. In this sense, our method provides a cognitively plausible way of building representations, consistent with the inherently re-constructive and associative nature of human memory. Using seven benchmark concept similarity tests we show that the mapped vectors not only implicitly encode multimodal information, but also outperform strong unimodal baselines and state-of-the-art multimodal methods, thus exhibiting more "human-like" judgments---particularly in zero-shot settings.

Motivation & Objective

• To develop a simple, efficient method for creating multimodal representations that integrate vision and language in a way consistent with human memory’s re-constructive and associative nature.
• To address the challenge of building rich, generalizable multimodal embeddings without requiring paired visual and textual data for every concept.
• To improve performance on semantic similarity tasks—especially in zero-shot settings—by leveraging cross-modal mappings that implicitly encode both modalities.
• To investigate whether a re-constructive, associative approach to multimodal representation learning yields more human-like semantic judgments than additive or concatenative methods.

Proposed method

• The method learns a language-to-vision mapping using a neural network (either feedforward or linear) that predicts visual features from word embeddings.
• The output of this mapping is used directly as a multimodal representation, combining linguistic input with reconstructed visual information.
• The model is trained to minimize reconstruction loss between predicted and ground-truth visual features, forcing the network to learn meaningful cross-modal associations.
• Multimodal embeddings are formed by concatenating the original word embedding with the mapped visual representation, preserving both linguistic and reconstructed visual information.
• The approach is applied to seven benchmark datasets, including Wordsim353, MEN, SimVerb-3500, and others, using human-annotated similarity scores as evaluation targets.
• Two variants are evaluated: MAP-NN (non-linear mapping) and MAP-Lin (linear mapping), with ablation studies comparing them to baselines like GloVe, CNN average, and feature concatenation (CONC).

Experimental results

Research questions

• RQ1 Can a language-to-vision mapping that reconstructs visual features from text produce more effective and human-like multimodal representations than unimodal or concatenative baselines?
• RQ2 Does the proposed re-constructive, associative method generalize better to zero-shot word pairs (where no paired image is available) than existing multimodal approaches?
• RQ3 Is the performance gain due to the learned mapping capturing more semantically relevant visual features than raw visual vectors, as suggested by ablation comparisons?
• RQ4 To what extent do the reconstructed visual features improve semantic similarity judgments, especially for concrete and visually grounded concepts?
• RQ5 Does the method’s cognitive plausibility—mimicking human memory’s re-constructive and associative nature—translate into more accurate, human-aligned similarity predictions?

Key findings

• The MAP-C NN variant achieved the highest Spearman correlation of 0.820 on the VisSim test set, outperforming all baselines including GloVe and CONC.
• On the Wordsim353-rel dataset, MAP-C NN achieved a correlation of 0.778 in the VIS region, significantly outperforming GloVe (0.688) and CONC (0.526), with a p-value ≈ 0.008 for the improvement.
• In zero-shot (ZS) settings, the method showed consistent gains across all test sets, with MAP-C NN achieving 0.745 on SimVerb-3500-ZS, outperforming GloVe (0.688) and CONC (0.526).
• The MAP-C NN model improved over the original visual feature concatenation (CONC) in 6 out of 7 test sets (p ≈ 0.06), indicating that mapped features are semantically richer than raw visual vectors.
• The method significantly outperformed simple concatenation (CONC) in all VIS regions (p ≈ 0.7 for CONC vs. p ≈ 0.008 for MAP-C NN), showing that association between modalities is more effective than additive fusion.
• The approach achieved state-of-the-art performance on all seven benchmark datasets, with the highest scores on MEN (0.800), VisSim (0.820), and SimVerb-3500 (0.785), supporting its claim of producing more 'human-like' judgments.