[Paper Review] Mapping Images to Psychological Similarity Spaces Using Neural Networks

Lucas Bechberger, Elektra Kypridemou|arXiv (Cornell University)|Apr 20, 2018

Image Retrieval and Classification Techniques22 references3 citations

TL;DR

This paper proposes a hybrid approach that uses psychological similarity ratings from human experiments to train neural networks to map images to conceptual spaces, enabling both psychological validity and generalization to unseen stimuli. The method achieves significantly lower RMSE on test data for correct mappings versus shuffled ones, demonstrating feasibility and generalization potential despite overfitting issues.

ABSTRACT

The cognitive framework of conceptual spaces bridges the gap between symbolic and subsymbolic AI by proposing an intermediate conceptual layer where knowledge is represented geometrically. There are two main approaches for obtaining the dimensions of this conceptual similarity space: using similarity ratings from psychological experiments and using machine learning techniques. In this paper, we propose a combination of both approaches by using psychologically derived similarity ratings to constrain the machine learning process. This way, a mapping from stimuli to conceptual spaces can be learned that is both supported by psychological data and allows generalization to unseen stimuli. The results of a first feasibility study support our proposed approach.

Motivation & Objective

• To bridge symbolic and subsymbolic AI by grounding conceptual spaces in psychological data.
• To address the limitation of purely machine learning methods, which lack psychological validity, and purely psychological methods, which lack generalization.
• To develop a neural network-based mapping from images to conceptual spaces that leverages human-derived similarity ratings.
• To evaluate whether psychological similarity spaces can guide the learning of generalizable image representations.
• To explore the feasibility of combining multidimensional scaling (MDS) derived similarity spaces with deep neural networks for image representation.

Proposed method

• Psychological similarity ratings from human experiments are used to generate a multidimensional scaling (MDS) space representing conceptual similarity.
• A pre-trained deep neural network (e.g., ResNet) extracts image features, producing a 2048-dimensional embedding per image.
• A linear regression model maps the image feature vectors to the MDS coordinates, using the MDS space as a target.
• The model is trained on a dataset of images with known similarity ratings, with loss measured by RMSE between predicted and actual MDS coordinates.
• A baseline comparison is performed using a randomly shuffled version of the MDS coordinates to test if the model learns meaningful structure.
• Data augmentation is applied to increase training diversity and mitigate overfitting.

Experimental results

Research questions

• RQ1 Can a neural network learn to map image features to a psychologically derived conceptual space with meaningful generalization?
• RQ2 Does using psychological similarity ratings as a target improve the generalization of image representations compared to random baselines?
• RQ3 How does the dimensionality of the MDS space affect the performance of the learned mapping?
• RQ4 To what extent does the model overfit, and can overfitting be mitigated with data augmentation and architectural constraints?
• RQ5 Is the learned mapping sensitive to domain-specific structure (e.g., color, shape, size) in the data?

Key findings

• The model achieved significantly lower RMSE on test data for the correct mapping compared to the shuffled mapping, indicating it learned meaningful structure.
• The performance gap between correct and shuffled mappings was consistent across different MDS dimensionalities, confirming the model captured psychological similarity structure.
• The model performed better in higher-dimensional MDS spaces, suggesting it can learn finer-grained similarity nuances when given more capacity.
• Overfitting was observed, with much lower RMSE on training data than test data, likely due to limited training data and high-dimensional feature vectors.
• Baseline models using shuffled MDS coordinates showed similar performance across dimensions, indicating the structure of the MDS space itself was not destroyed by shuffling.
• The results support the feasibility of combining psychological data with neural networks to learn generalizable, psychologically grounded image representations.