[论文解读] "Genlangs" and Zipf's Law: Do languages generated by ChatGPT statistically look human?

Justin Diamond|arXiv (Cornell University)|Mar 31, 2023

Topic Modeling被引用 4

一句话总结

本研究探讨了由ChatGPT生成的AI构造语言（'genlangs'）是否表现出自然人类语言的统计特性，重点关注其是否符合齐夫定律。通过对三种AI生成语言（Voxphera、Vivenzia、Lumivoxa）的语料库进行统计语言学分析，论文发现genlangs与齐夫定律高度吻合，表明其在统计上与人类语言相似，支持了在人类指导下的AI创建完全功能化的构造语言的可行性。

ABSTRACT

OpenAI's GPT-4 is a Large Language Model (LLM) that can generate coherent constructed languages, or "conlangs," which we propose be called "genlangs" when generated by Artificial Intelligence (AI). The genlangs created by ChatGPT for this research (Voxphera, Vivenzia, and Lumivoxa) each have unique features, appear facially coherent, and plausibly "translate" into English. This study investigates whether genlangs created by ChatGPT follow Zipf's law. Zipf's law approximately holds across all natural and artificially constructed human languages. According to Zipf's law, the word frequencies in a text corpus are inversely proportional to their rank in the frequency table. This means that the most frequent word appears about twice as often as the second most frequent word, three times as often as the third most frequent word, and so on. We hypothesize that Zipf's law will hold for genlangs because (1) genlangs created by ChatGPT fundamentally operate in the same way as human language with respect to the semantic usefulness of certain tokens, and (2) ChatGPT has been trained on a corpora of text that includes many different languages, all of which exhibit Zipf's law to varying degrees. Through statistical linguistics, we aim to understand if LLM-based languages statistically look human. Our findings indicate that genlangs adhere closely to Zipf's law, supporting the hypothesis that genlangs created by ChatGPT exhibit similar statistical properties to natural and artificial human languages. We also conclude that with human assistance, AI is already capable of creating the world's first fully-functional genlang, and we call for its development.

研究动机与目标

• 确定大型语言模型生成的构造语言（genlangs）是否遵循齐夫定律，即自然人类语言的统计特征。
• 评估由ChatGPT创建的genlangs是否表现出与人类语言及人工构造语言相同的语言规律性。
• 评估在人类监督下，AI生成完全功能性构造语言的潜力。
• 探究大型语言模型的训练数据分布——富含具有齐夫频率模式的人类语言——是否导致genlangs继承这些统计特性。

提出的方法

• 通过提示工程使用GPT-4生成三种构造语言（Voxphera、Vivenzia、Lumivoxa），以生成连贯、词汇一致且具备翻译能力的语言系统。
• 为每种genlang整理了10,000个词以上的文本语料库，确保语言连贯性和词汇多样性。
• 从每个语料库中提取词频分布，并按频率排序，以分析其对齐夫定律的符合程度。
• 生成词频与词频排名的对数-对数图，并使用线性回归的R²值量化拟合优度。
• 将回归线的斜率与理论齐夫定律斜率-1进行比较，评估genlangs与预期反比关系的接近程度。
• 通过R²值和频率分布的视觉检查评估统计显著性，以确定genlangs是否遵循幂律模式。

实验结果

研究问题

• RQ1 由ChatGPT生成的AI构造语言（genlangs）是否遵循齐夫定律，即人类语言的统计特征？
• RQ2 genlangs中词频分布在多大程度上反映了齐夫定律预测的反比关系？
• RQ3 genlangs的统计特性是否与自然语言及人工人类语言相似，特别是在词频缩放方面？
• RQ4 LLM的训练数据（包含大量具有齐夫分布的人类语言）是否导致genlangs继承这一特性？
• RQ5 对齐夫定律的符合程度对AI生成语言的'人类相似性'感知意味着什么？

主要发现

• genlangs Voxphera、Vivenzia和Lumivoxa均表现出对齐夫定律的强烈遵循，三者的R²值均超过0.95。
• 每种genlang的对数-对数词频排名回归线斜率接近-1，表明其分布符合齐夫定律的幂律特性。
• 对频率分布的视觉检查确认，最常用词的出现频率显著高于较不常用的词，符合预期的反比关系。
• 结果表明，由ChatGPT生成的genlangs继承了人类语言的统计规律性，即使它们并非人类所创。
• 本研究得出结论：genlangs在词频分布方面与人类语言在统计上无法区分，支持其作为功能性构造语言的潜力。
• 在人类协助下，本文认为AI已具备创建完全功能化、类人构造语言的能力。