Skip to main content
QUICK REVIEW

[论文解读] Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loop

Martin Briesch, Dominik Sobania|arXiv (Cornell University)|Nov 28, 2023
Topic ModelingComputer Science被引用 8
一句话总结

该论文在实证层面分析了用于大语言模型的自耗训练循环,使用新颖的逻辑表达式数据集,展示初始正确性与多样性提升,随后多样性崩溃,崩溃速度取决于数据循环和合成数据比例。

ABSTRACT

Large Language Models (LLM) are already widely used to generate content for a variety of online platforms. As we are not able to safely distinguish LLM-generated content from human-produced content, LLM-generated content is used to train the next generation of LLMs, giving rise to a self-consuming training loop. From the image generation domain we know that such a self-consuming training loop reduces both quality and diversity of images finally ending in a model collapse. However, it is unclear whether this alarming effect can also be observed for LLMs. Therefore, we present the first study investigating the self-consuming training loop for LLMs. Further, we propose a novel method based on logic expressions that allows us to unambiguously verify the correctness of LLM-generated content, which is difficult for natural language text. We find that the self-consuming training loop produces correct outputs, however, the output declines in its diversity depending on the proportion of the used generated data. Fresh data can slow down this decline, but not stop it. Given these concerning results, we encourage researchers to study methods to negate this process.

研究动机与目标

  • Motivate the study of self-consuming training loops where LLMs are trained on their own outputs.
  • Introduce a logical-expression dataset to enable exact correctness verification of generated outputs.
  • Evaluate how different data-cycle designs affect correctness and diversity over generations.
  • Provide metrics and insights to guide safe use of self-generated data in future LLM training.

提出的方法

  • Use a GPT-style model (nanoGPT) with 10.6M parameters to simulate self-consuming training loops.
  • Create a 10k true-logical-expression dataset as the verification ground truth.
  • Train models for up to 50 generations with four data-cycle designs (full synthetic, balanced, incremental, expanding).
  • Sample 10k expressions per generation with autoregressive generation (temperature 0.8, up to 200 tokens per expression).
  • Measure correctness by syntactic validity and boolean evaluation; measure diversity with Levenshtein diversity on tokenized expressions.
  • Employ a fixed training setup: 6-layer, 6-head, 384-dim, context 256, batch 64, dropout 0.2, 5000 iterations, learning rate 1e-3 decaying to 1e-4, 90/10 train/validation split.

实验结果

研究问题

  • RQ1Does self-consuming training improve the semantic correctness of LLM outputs over generations?
  • RQ2How does the composition of data cycles (full synthetic, balanced, incremental, expanding) affect correctness and diversity over generations?
  • RQ3What is the impact of the amount of synthetic data introduced per generation on output diversity?
  • RQ4Can we verify LLM-generated outputs rigorously using a logic-expression dataset?
  • RQ5Is there a point at which diversity collapses in self-consuming training loops?

主要发现

  • Correctness improves over generations across data cycles, with faster improvement in full synthetic cycles and slower progress in others.
  • Diversity initially increases but then declines across generations; full synthetic and many cycles eventually collapse diversity to a single point, with expanding cycling showing the slowest degradation.
  • In the incremental cycle, increasing the synthetic data share accelerates diversity loss, with larger lambda causing faster collapse; lambda=0.1 can delay collapse somewhat.
  • Expanding data cycle shows no decline in diversity by generation 50, but the authors expect eventual collapse without intervention.
  • A logical-expression verification approach provides a provable correctness check for generated outputs, complementing traditional similarity metrics.

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。