[论文解读] Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges

J. K. Becker, Jan Philip Wahle+2 more|arXiv (Cornell University)|May 24, 2024

Advanced Text Analysis Techniques被引用 5

一句话总结

本文对文本生成领域的244篇工作（2017–2024）进行了综述，归类任务、评估指标，并总结九个共同挑战，提出未来研究建议。

ABSTRACT

Text generation has become more accessible than ever, and the increasing interest in these systems, especially those using large language models, has spurred an increasing number of related publications. We provide a systematic literature review comprising 244 selected papers between 2017 and 2024. This review categorizes works in text generation into five main tasks: open-ended text generation, summarization, translation, paraphrasing, and question answering. For each task, we review their relevant characteristics, sub-tasks, and specific challenges (e.g., missing datasets for multi-document summarization, coherence in story generation, and complex reasoning for question answering). Additionally, we assess current approaches for evaluating text generation systems and ascertain problems with current metrics. Our investigation shows nine prominent challenges common to all tasks and sub-tasks in recent text generation publications: bias, reasoning, hallucinations, misuse, privacy, interpretability, transparency, datasets, and computing. We provide a detailed analysis of these challenges, their potential solutions, and which gaps still require further engagement from the community. This systematic literature review targets two main audiences: early career researchers in natural language processing looking for an overview of the field and promising research directions, as well as experienced researchers seeking a detailed view of tasks, evaluation methodologies, open challenges, and recent mitigation strategies.

研究动机与目标

• 提供2017–2024年最新文本生成研究的全面概览。
• 识别并分类文本生成中的主要任务与子任务。
• 评估方法及其局限性。
• 突出普遍存在的挑战并提出潜在缓解方向。
• 为自然语言处理（NLP）和自然语言生成（NLG）领域的初级与资深研究人员提供指导。

提出的方法

• 遵循 Kitchenham 与 Charters 的指南进行系统文献综述。
• 通过时间和引用基准对1669篇出版物进行自动筛选，然后进行人工相关性评估。
• 对每篇相关论文按任务、子任务、数据集、指标和挑战进行标注和分类。
• 通过引用、专家意见和 Google Scholar 搜索发现的辅助论文来扩展数据集。
• 在开放获取的仓库中公开分享方法论和数据集元数据。

Figure 1. Taxonomy of the main tasks in text generation and their respective challenges.

实验结果

研究问题

• RQ1 文本生成任务的组成是什么，以及主要的子任务有哪些？
• RQ2 文本生成系统如何被评估，当前评估指标的局限性是什么？
• RQ3 文本生成领域存在哪些开放挑战？
• RQ4 文本生成领域出现了哪些突出的研究方向？

主要发现

• 确定的五大任务：开放式文本生成、摘要、翻译、改写和问答。
• 跨任务的九大共性挑战：偏见、推理、幻觉、滥用、隐私、可解释性、透明度、数据集和计算。
• 当前评估指标在无模型与基于模型的方法上存在局限性和空白。
• 开放领域的开放式生成因闭源模型而在可重复性与开放性方面存在问题。
• 对话、多文档和长上下文摘要呈现出独特的一致性与忠实性挑战。
• 翻译在低资源语言和训练/测试数据不匹配方面存在困难，回译作为缓解手段。

Figure 2. Pipeline of this systematic review. Red-colored boxes show papers we excluded; Green-colored boxes show papers we added.