[Paper Review] "Genlangs" and Zipf's Law: Do languages generated by ChatGPT statistically look human?

Justin Diamond|arXiv (Cornell University)|Mar 31, 2023

Topic Modeling4 citations

TL;DR

This study investigates whether AI-generated constructed languages ('genlangs') created by ChatGPT exhibit statistical properties of natural human languages, focusing on adherence to Zipf's law. Using statistical linguistics on corpora from three AI-generated languages (Voxphera, Vivenzia, Lumivoxa), the paper finds that genlangs closely follow Zipf's law, suggesting they statistically resemble human languages and supporting the feasibility of AI-created fully functional conlangs with human guidance.

ABSTRACT

OpenAI's GPT-4 is a Large Language Model (LLM) that can generate coherent constructed languages, or "conlangs," which we propose be called "genlangs" when generated by Artificial Intelligence (AI). The genlangs created by ChatGPT for this research (Voxphera, Vivenzia, and Lumivoxa) each have unique features, appear facially coherent, and plausibly "translate" into English. This study investigates whether genlangs created by ChatGPT follow Zipf's law. Zipf's law approximately holds across all natural and artificially constructed human languages. According to Zipf's law, the word frequencies in a text corpus are inversely proportional to their rank in the frequency table. This means that the most frequent word appears about twice as often as the second most frequent word, three times as often as the third most frequent word, and so on. We hypothesize that Zipf's law will hold for genlangs because (1) genlangs created by ChatGPT fundamentally operate in the same way as human language with respect to the semantic usefulness of certain tokens, and (2) ChatGPT has been trained on a corpora of text that includes many different languages, all of which exhibit Zipf's law to varying degrees. Through statistical linguistics, we aim to understand if LLM-based languages statistically look human. Our findings indicate that genlangs adhere closely to Zipf's law, supporting the hypothesis that genlangs created by ChatGPT exhibit similar statistical properties to natural and artificial human languages. We also conclude that with human assistance, AI is already capable of creating the world's first fully-functional genlang, and we call for its development.

Motivation & Objective

• To determine whether large language model-generated constructed languages (genlangs) follow Zipf's law, a statistical hallmark of natural human languages.
• To assess whether genlangs created by ChatGPT exhibit the same linguistic regularities observed in human and artificial conlangs.
• To evaluate the potential of AI, with human oversight, to produce fully functional constructed languages.
• To investigate whether the training data distribution of LLMs—rich in human languages with Zipfian frequency patterns—leads to genlangs that inherit these statistical properties.

Proposed method

• The study generated three constructed languages (Voxphera, Vivenzia, Lumivoxa) using GPT-4 via prompt engineering to produce coherent, lexically consistent, and translation-capable linguistic systems.
• Text corpora of 10,000+ words were compiled for each genlang, ensuring linguistic coherence and lexical diversity.
• Word frequency distributions were extracted from each corpus and ranked by frequency to analyze their adherence to Zipf's law.
• A log-log plot of word rank versus frequency was generated, and the goodness of fit was quantified using linear regression R² values.
• The study compared the slope of the regression line to the theoretical Zipfian slope of -1, assessing how closely genlangs matched the expected inverse proportionality.
• Statistical significance was evaluated using R² and visual inspection of frequency distributions to determine if genlangs followed the power-law pattern.

Experimental results

Research questions

• RQ1 Do AI-generated constructed languages (genlangs) created by ChatGPT follow Zipf's law, a statistical signature of human language?
• RQ2 To what extent do the frequency distributions of words in genlangs mirror the inverse proportionality predicted by Zipf's law?
• RQ3 Do the statistical properties of genlangs resemble those of natural and artificial human languages, particularly in terms of word frequency scaling?
• RQ4 Can the training data of LLMs, which includes many human languages with Zipfian distributions, lead to genlangs that inherit this property?
• RQ5 What does the adherence to Zipf's law imply about the perceived 'human-likeness' of AI-generated languages?

Key findings

• The genlangs Voxphera, Vivenzia, and Lumivoxa exhibited strong adherence to Zipf's law, with R² values exceeding 0.95 for all three languages.
• The slope of the log-log frequency-rank regression for each genlang was close to -1, indicating a power-law distribution consistent with Zipf's law.
• Visual inspection of frequency distributions confirmed that the most frequent words occurred significantly more often than less frequent ones, following the expected inverse proportionality.
• The results suggest that genlangs generated by ChatGPT inherit the statistical regularities of human language, even though they are not human-produced.
• The study concludes that genlangs are statistically indistinguishable from human languages in terms of word frequency distribution, supporting their potential as functional conlangs.
• With human assistance, the paper argues that AI is already capable of creating fully functional, human-like constructed languages.