[Paper Review] Evaluating Multimodal Representations on Sentence Similarity: vSTS, Visual Semantic Textual Similarity Dataset

Oier López de Lacalle, Aitor Soroa+1 more|arXiv (Cornell University)|Sep 11, 2018

Topic Modeling18 references3 citations

TL;DR

This paper introduces vSTS, a multimodal dataset combining sentence pairs with human-annotated similarity scores (0–5) and corresponding images to evaluate visual-augmented sentence representations. Using a combination of text-only models (e.g., Decomposable Attention Model) and image representations (ResNet-50), the study demonstrates that visual information significantly improves sentence similarity prediction, achieving a Pearson correlation of 0.78 on the test set when combining the best text and image models.

ABSTRACT

In this paper we introduce vSTS, a new dataset for measuring textual similarity of sentences using multimodal information. The dataset is comprised by images along with its respectively textual captions. We describe the dataset both quantitatively and qualitatively, and claim that it is a valid gold standard for measuring automatic multimodal textual similarity systems. We also describe the initial experiments combining the multimodal information.

Motivation & Objective

• To create a standardized benchmark for evaluating how visual information enhances sentence representation models in semantic similarity tasks.
• To investigate whether image representations alone can predict sentence similarity.
• To examine whether combining image and text representations improves performance over text-only models.

Proposed method

• The vSTS dataset is derived from the STS benchmark's 'STS-images' subset, comprising 829 sentence pairs with human similarity scores and corresponding images.
• Image representations are extracted using a pretrained ResNet-50 model, while text representations are generated via word overlap, GloVe embeddings (centroid averaging), and the Decomposable Attention Model (DAM).
• Similarity scores are computed using cosine similarity between sentence and image embeddings.
• Model combinations are tested using addition, multiplication, and linear regression to fuse text and image predictions.
• The dataset is split into development and test sets (50% each), preserving similarity score distributions.
• Linear regression parameters are tuned via 10-fold cross-validation on the development set.

Experimental results

Research questions

• RQ1 Can image representations alone predict sentence similarity with reasonable accuracy?
• RQ2 Does combining image and text representations lead to improved performance on sentence similarity compared to text-only models?
• RQ3 How do different text-based models (e.g., word overlap, GloVe, DAM) perform when augmented with visual features?

Key findings

• The image-only ResNet-50 model achieved a Pearson correlation of 0.61 on the test set, indicating that visual representations alone carry meaningful similarity information.
• The text-only Decomposable Attention Model (DAM) achieved the highest performance among single models, with a test correlation of 0.69.
• Combining DAM with ResNet-50 via linear regression yielded the best result, achieving a test Pearson correlation of 0.78.
• Addition and multiplication of predictions also improved performance, with the best combination (DAM + ResNet-50) reaching 0.78 using all fusion methods.
• The dataset shows a strong bias toward low similarity scores (159 out of 819 instances scored 0), but still maintains sufficient variability for meaningful evaluation.
• Results confirm that visual and textual representations are complementary, with visual input enhancing text-based inference in multimodal sentence similarity tasks.