[Paper Review] Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges
This paper surveys 244 works (2017–2024) on text generation, categorizing tasks, evaluating metrics, and nine shared challenges, with recommendations for future research.
Text generation has become more accessible than ever, and the increasing interest in these systems, especially those using large language models, has spurred an increasing number of related publications. We provide a systematic literature review comprising 244 selected papers between 2017 and 2024. This review categorizes works in text generation into five main tasks: open-ended text generation, summarization, translation, paraphrasing, and question answering. For each task, we review their relevant characteristics, sub-tasks, and specific challenges (e.g., missing datasets for multi-document summarization, coherence in story generation, and complex reasoning for question answering). Additionally, we assess current approaches for evaluating text generation systems and ascertain problems with current metrics. Our investigation shows nine prominent challenges common to all tasks and sub-tasks in recent text generation publications: bias, reasoning, hallucinations, misuse, privacy, interpretability, transparency, datasets, and computing. We provide a detailed analysis of these challenges, their potential solutions, and which gaps still require further engagement from the community. This systematic literature review targets two main audiences: early career researchers in natural language processing looking for an overview of the field and promising research directions, as well as experienced researchers seeking a detailed view of tasks, evaluation methodologies, open challenges, and recent mitigation strategies.
Motivation & Objective
- Provide a comprehensive overview of recent text generation research (2017–2024).
- Identify and categorize main tasks and sub-tasks in text generation.
- Assess evaluation methodologies and their limitations.
- Highlight pervasive challenges and propose potential mitigation directions.
- Offer guidance for early-career and experienced researchers in NLP and NLG.
Proposed method
- Conduct systematic literature review following Kitchenham and Charters guidelines.
- Automated filtering of 1669 publications via temporal and citation-based criteria, then manual relevance assessment.
- Annotate and categorize each relevant paper by task, sub-task, datasets, metrics, and challenges.
- Augment dataset with auxiliary papers identified through citations, expert input, and Google Scholar searches.
- Publicly share methodology and dataset metadata in an open-access repository.
Experimental results
Research questions
- RQ1What constitutes the task of text generation and what are the main sub-tasks?
- RQ2How are text generation systems evaluated and what are the limitations of current metrics?
- RQ3What are the open challenges in text generation?
- RQ4What prominent research directions emerge in text generation?
Key findings
- Five main tasks identified: open-ended text generation, summarization, translation, paraphrasing, and question answering.
- Nine cross-cutting challenges common across tasks: bias, reasoning, hallucinations, misuse, privacy, interpretability, transparency, datasets, and computing.
- Current evaluation metrics face limitations and gaps across model-free and model-based approaches.
- Open-domain open-ended generation faces reproducibility and openness issues due to closed-source models.
- Dialogue, multi-document, and long-context summarization present distinct coherence and faithfulness challenges.
- Translation struggles with low-resource languages and train/test data mismatch, with back-translation as a mitigation.
Better researchstarts right now
From paper design to paper writing, dramatically reduce your research time.
No credit card · Free plan available
This review was created by AI and reviewed by human editors.