[论文解读] The Vector Grounding Problem

Dimitri Coelho Mollo, Raphaël Millière|arXiv (Cornell University)|Apr 4, 2023

Language and cultural evolution被引用 35

一句话总结

这篇论文区分了五种接地概念，主张指称性接地支持向量接地问题，并认为通过 RLHF 启用的 LLMs 可以在不需要多模态或具身性的情况下对意义进行接地。

ABSTRACT

Large language models (LLMs) produce seemingly meaningful outputs, yet they are trained on text alone without direct interaction with the world. This leads to a modern variant of the classical symbol grounding problem in AI: can LLMs' internal states and outputs be about extra-linguistic reality, independently of the meaning human interpreters project onto them? We argue that they can. We first distinguish referential grounding -- the connection between a representation and its worldly referent -- from other forms of grounding and argue it is the only kind essential to solving the problem. We contend that referential grounding is achieved when a system's internal states satisfy two conditions derived from teleosemantic theories of representation: (1) they stand in appropriate causal-informational relations to the world, and (2) they have a history of selection that has endowed them with the function of carrying this information. We argue that LLMs can meet both conditions, even without multimodality or embodiment.

研究动机与目标

• 区分内部表征如何在生物系统或人工系统中实现接地。
• 确认五种接地概念并澄清哪一种是向量接地问题的核心。
• 论证通过 RLHF 启用的 LLMs 具有实现接地所需的因果-历史关系。
• 评估多模态性或具身性是否是人工系统进行指称性接地的必要或充分条件。

提出的方法

• 调查并界定五种接地概念：指称性、感知运动、关系性、交际性、认知性。
• 解释经典 AI 中的符号接地问题并将其转化为 LLMs 的向量化设定。
• 引入 Octopus Test 作为解决 LLMs 接地挑战的直觉推动工具。
• 论证指称性接地对 LLMs 的接地至关重要，并讨论 RLHF 与上下文学习如何贡献。
• 评估多模态性和具身性在人工系统接地中的必要性或充分性。

Figure 1: Five notions of grounding. A. Referential grounding : a lexical representation ( <DOG> ) is connected to its worldly referent (a dog). B. Sensorimotor grounding : a lexical representation ( <DOG> ) is connected to a sensory representation (image of a dog). C. Relational grounding : a lexic

实验结果

研究问题

• RQ1 不同的接地概念有哪些，哪一个是将内部表征在 LLMs 中接地的核心？
• RQ2 大型预训练模型是否能够通过接地机制实现内在意义，特别是通过 RLHF 与上下文学习？
• RQ3 多模态性和具身性是否是人工系统进行指称性接地的必要条件？

主要发现

• LLMs 面临符号接地问题的向量化类似物，因此需要接地方面的考虑。
• 指称性接地被确认为 LLMs 接地的核心概念，将内部表征与世界参照物相连接。
• RLHF 与上下文学习提供了能够建立对世界所需的因果-历史关系的特征。
• 认为多模态性和具身性既非必要条件也非指称性接地的充分条件。
• 论文强调，具备 grounding 的内部状态可能对更广泛的认知能力至关重要，但本身并不足以实现理解或自主性。

Figure 2: Sensorigraph and S-model. Left: the sensorigraph records the host’s sensory input and motor input over a lifetime. Right: an autoregressive S-model is trained on sensorigraph data by predicting the embedding for the next timestep in the sensorimotor stream sequence.