[Paper Review] Gromov-Wasserstein unsupervised alignment reveals structural correspondences between the color similarity structures of humans and large language models

Genji Kawakita, Ariel Zeleznikow-Johnston+2 more|arXiv (Cornell University)|Aug 8, 2023

Computational and Text Analysis Methods28 references9 citations

TL;DR

The paper uses unsupervised alignment via Gromov-Wasserstein optimal transport to compare color similarity structures of humans and two GPT models (GPT-3.5 and GPT-4), finding GPT-4’s structure aligns closely with color-neurotypical humans at a fine-item level.

ABSTRACT

Large Language Models (LLMs), such as the General Pre-trained Transformer (GPT), have shown remarkable performance in various cognitive tasks. However, it remains unclear whether these models have the ability to accurately infer human perceptual representations. Previous research has addressed this question by quantifying correlations between similarity response patterns of humans and LLMs. Correlation provides a measure of similarity, but it relies pre-defined item labels and does not distinguish category- and item- level similarity, falling short of characterizing detailed structural correspondence between humans and LLMs. To assess their structural equivalence in more detail, we propose the use of an unsupervised alignment method based on Gromov-Wasserstein optimal transport (GWOT). GWOT allows for the comparison of similarity structures without relying on pre-defined label correspondences and can reveal fine-grained structural similarities and differences that may not be detected by simple correlation analysis. Using a large dataset of similarity judgments of 93 colors, we compared the color similarity structures of humans (color-neurotypical and color-atypical participants) and two GPT models (GPT-3.5 and GPT-4). Our results show that the similarity structure of color-neurotypical participants can be remarkably well aligned with that of GPT-4 and, to a lesser extent, to that of GPT-3.5. These results contribute to the methodological advancements of comparing LLMs with human perception, and highlight the potential of unsupervised alignment methods to reveal detailed structural correspondences. This work has been published in Scientific Reports, DOI: https://doi.org/10.1038/s41598-024-65604-1.

Motivation & Objective

• Assess whether LLMs infer human perceptual color representations beyond correlation analysis.
• Evaluate the fine-item level structural alignment between human color similarity structures and LLMs.
• Contrast GPT-4 and GPT-3.5 against color-space baselines (RGB, LAB) and color-atypical participants.
• Demonstrate the utility of GWOT-based unsupervised alignment for examining representational equivalence.

Proposed method

• Collect color similarity judgments for 93 colors from color-neurotypical (n=426) and color-atypical (n=207) human participants.
• Obtain color similarity judgments from GPT-3.5 (gpt-3.5-turbo) and GPT-4 (gpt-4-0314) using hex-encoded color prompts and 5 trials per pair, averaging results.
• Create RGB and LAB color-space similarity matrices using Euclidean distance and delta_E_cie2000 respectively.
• Compute conventional RSA correlations between similarity matrices.
• Apply Gromov-Wasserstein optimal transport (GWOT) to align similarity structures without assuming label correspondence; optimize with entropy regularization (epsilon in [1e-4,1e-1]).
• Evaluate alignment via top-1 and top-k matching rates using the GW transport plan.

Figure 1: Schematic of unsupervised alignment. (a) Unsupervised alignment of similarity structures without external labels, based only on similarity relations. (b) Evaluation of unsupervised alignment using external labels. (c) Schematic of Gromov-Wasserstein optimal transport. The elements of matri

Experimental results

Research questions

• RQ1 Can unsupervised GWOT alignment reveal fine-item level correspondences between human color similarity structures and LLMs, beyond traditional correlations?
• RQ2 Which models (GPT-4, GPT-3.5, RGB, LAB) most closely align with human color perception structurally?
• RQ3 Do color-atypical participants align with LLMs or color-space models similarly to color-neurotypical participants?
• RQ4 Does GPT-4’s color structure align more with humans than GPT-3.5 when considering unsupervised alignment?

Key findings

• Color-neurotypical human color structures align remarkably well with GPT-4 under unsupervised GWOT alignment (top-1 ~80%), near the human-human alignment (~86%).
• GPT-3.5 shows lower alignment (top-1 ~31%) compared with GPT-4.
• Color-space models (RGB: ~3.8% top-1; LAB: ~7.0% top-1) show little unsupervised alignment with human color structures, despite moderate RSA correlations (RGB 0.60, LAB 0.71).
• GPT-4’s unsupervised alignment outperforms GPT-3.5 and color-space baselines, revealing nuanced structural similarity not captured by correlation alone.
• Color-atypical participants exhibit lower correlations and poorer alignment, indicating differing color similarity structures from neurotypical humans and LLMs.
• Visualization of aligned embeddings shows similar colors clustering together across neurotypical humans and GPT-4.

Figure 2: Similarity matrices Displayed are the similarity matrices of 93 colors from two non-overlapping color-neurotypical participants groups, one color-atypical participants group, GPT-4, GPT-3.5, and the RGB and LAB color space models. All matrices are normalized to have values between 0 and 1.