[论文解读] Gromov-Wasserstein unsupervised alignment reveals structural correspondences between the color similarity structures of humans and large language models

Genji Kawakita, Ariel Zeleznikow-Johnston+2 more|arXiv (Cornell University)|Aug 8, 2023

Computational and Text Analysis Methods参考文献 28被引用 9

一句话总结

本论文通过使用基于Gromov-Wasserstein最优传输的无监督对齐，比较人类与两种GPT模型（GPT-3.5与GPT-4）的颜色相似性结构，发现GPT-4的结构在精细项层面与色觉神经正常人高度对齐。

ABSTRACT

Large Language Models (LLMs), such as the General Pre-trained Transformer (GPT), have shown remarkable performance in various cognitive tasks. However, it remains unclear whether these models have the ability to accurately infer human perceptual representations. Previous research has addressed this question by quantifying correlations between similarity response patterns of humans and LLMs. Correlation provides a measure of similarity, but it relies pre-defined item labels and does not distinguish category- and item- level similarity, falling short of characterizing detailed structural correspondence between humans and LLMs. To assess their structural equivalence in more detail, we propose the use of an unsupervised alignment method based on Gromov-Wasserstein optimal transport (GWOT). GWOT allows for the comparison of similarity structures without relying on pre-defined label correspondences and can reveal fine-grained structural similarities and differences that may not be detected by simple correlation analysis. Using a large dataset of similarity judgments of 93 colors, we compared the color similarity structures of humans (color-neurotypical and color-atypical participants) and two GPT models (GPT-3.5 and GPT-4). Our results show that the similarity structure of color-neurotypical participants can be remarkably well aligned with that of GPT-4 and, to a lesser extent, to that of GPT-3.5. These results contribute to the methodological advancements of comparing LLMs with human perception, and highlight the potential of unsupervised alignment methods to reveal detailed structural correspondences. This work has been published in Scientific Reports, DOI: https://doi.org/10.1038/s41598-024-65604-1.

研究动机与目标

• 评估LLM是否在相关性分析之外推断人类感知颜色表示。
• 评估人类颜色相似性结构与LLM之间的精细项层级结构对齐。
• 将GPT-4与GPT-3.5与颜色空间基线（RGB、LAB）及色觉异常参与者进行对比。
• 证明基于GWOT的无监督对齐在检验表征等价性方面的实用性。

提出的方法

• 收集来自色觉神经正常人（n=426）和色觉异常人群（n=207）的93种颜色的颜色相似性判断。
• 使用十六进制颜色提示从GPT-3.5（gpt-3.5-turbo）和GPT-4（gpt-4-0314）获得颜色相似性判断，对每对颜色进行5次试验并取均值。
• 使用欧氏距离创建RGB颜色空间相似性矩阵；使用delta_E_cie2000创建LAB颜色空间相似性矩阵。
• 在相似性矩阵之间计算传统的RSA相关性。
• 应用Gromov-Wasserstein最优传输（GWOT）在不假设标签对应的情况下对齐相似性结构；用熵正则化（epsilon在[1e-4,1e-1]之间）进行优化。
• 通过GW传输计划的top-1和top-k匹配率来评估对齐效果。

Figure 1: Schematic of unsupervised alignment. (a) Unsupervised alignment of similarity structures without external labels, based only on similarity relations. (b) Evaluation of unsupervised alignment using external labels. (c) Schematic of Gromov-Wasserstein optimal transport. The elements of matri

实验结果

研究问题

• RQ1 无监督GWOT对齐是否能揭示人类颜色相似性结构与LLM之间在精细项层面的对应关系，超越传统相关性？
• RQ2 哪些模型（GPT-4、GPT-3.5、RGB、LAB）在结构上最接近人类色觉感知？
• RQ3 颜色异常参与者与LLM或颜色空间模型的对齐是否与色觉神经正常人参与者相似？
• RQ4 在考虑无监督对齐时，GPT-4的颜色结构是否比GPT-3.5更接近人类？

主要发现

• 在无监督的GWOT对齐下，色觉神经正常人的颜色结构与GPT-4高度对齐（top-1约80%），接近人-人对齐（约86%）。
• GPT-3.5的对齐较低（top-1约31%）相比GPT-4。
• 颜色空间模型（RGB：top-1约3.8%；LAB：top-1约7.0%）在无监督对齐方面与人类颜色结构几乎无关联，尽管RSA相关性中等（RGB 0.60，LAB 0.71）。
• GPT-4的无监督对齐胜过GPT-3.5和颜色空间基线，揭示仅凭相关性无法捕捉的细致结构相似性。
• 颜色异常参与者显示较低的相关性和较差的对齐，表明其颜色相似性结构与神经正常人和LLM不同。
• 对齐嵌入的可视化显示，在神经正常人和GPT-4之间，类似颜色聚集在一起。

Figure 2: Similarity matrices Displayed are the similarity matrices of 93 colors from two non-overlapping color-neurotypical participants groups, one color-atypical participants group, GPT-4, GPT-3.5, and the RGB and LAB color space models. All matrices are normalized to have values between 0 and 1.