[Paper Review] Automatic Related Work Generation: A Meta Study

Xiangci Li, Jessica Ouyang|arXiv (Cornell University)|Jan 6, 2022

Advanced Text Analysis Techniques4 citations

TL;DR

This meta-study critically evaluates automatic related work generation in scientific NLP, analyzing problem formulations, datasets, methods, and evaluation across recent works. It identifies key challenges like lack of standardization, reliance on sentence-level units, factuality issues, and limited domain scope, while advocating for knowledge-enhanced, multi-document modeling and standardized benchmarks to advance the field beyond current fluency-focused, narrow-scope systems.

ABSTRACT

Academic research is an exploration activity to solve problems that have never been resolved before. By this nature, each academic research work is required to perform a literature review to distinguish its novelties that have not been addressed by prior works. In natural language processing, this literature review is usually conducted under the "Related Work" section. The task of automatic related work generation aims to automatically generate the "Related Work" section given the rest of the research paper and a list of cited papers. Although this task was proposed over 10 years ago, it received little attention until very recently, when it was cast as a variant of the scientific multi-document summarization problem. However, even today, the problems of automatic related work and citation text generation are not yet standardized. In this survey, we conduct a meta-study to compare the existing literature on related work generation from the perspectives of problem formulation, dataset collection, methodological approach, performance evaluation, and future prospects to provide the reader insight into the progress of the state-of-the-art studies, as well as and how future studies can be conducted. We also survey relevant fields of study that we suggest future work to consider integrating.

Motivation & Objective

• To provide a comprehensive, comparative analysis of existing approaches to automatic related work generation in scientific NLP.
• To identify critical gaps and inconsistencies in problem formulation, dataset construction, and evaluation practices across prior works.
• To highlight methodological limitations such as over-reliance on sentence-level units and partial paper representations (e.g., abstracts).
• To address the lack of standardization in task definition, evaluation metrics, and shared benchmarks.
• To advocate for integrating external knowledge, improving factuality, and extending beyond computational linguistics domains.

Proposed method

• Conducts a meta-study comparing 10+ recent works on automatic related work generation across problem formulation, dataset collection, methodology, evaluation, and future directions.
• Analyzes methodological approaches including neural sequence models, multi-document encoders, and integration of citation networks and discourse signals.
• Evaluates performance using automatic metrics (e.g., ROUGE) and highlights the lack of human evaluation for factuality and coherence.
• Proposes a framework for improving data collection by combining manual and automatic techniques to enhance efficiency and quality.
• Recommends incorporating external supervision signals such as citation function labels, salience scores, and knowledge bases to improve factual consistency.
• Stresses the need for modeling full-text representations and developing end-to-end systems that generate coherent, factual, and succinct full related work sections.

Experimental results

Research questions

• RQ1 How do existing works differ in their formulation of the automatic related work generation task?
• RQ2 What are the dominant data collection strategies, and how do they impact model generalization and evaluation?
• RQ3 To what extent do current methods rely on simplistic assumptions such as sentence-level units or abstract-only representations?
• RQ4 How effective are current evaluation practices in measuring factual consistency, coherence, and informativeness?
• RQ5 What role can external knowledge, citation networks, and discourse analysis play in improving the quality of generated related work sections?

Key findings

• Most studies rely on abstracts or conclusions as proxies for full papers, which oversimplifies the representation of cited works and limits performance.
• The majority of systems treat sentences as the basic unit of generation, despite evidence that citation contexts often span multiple sentences or involve multiple papers.
• There is a significant lack of standardization in task definition, datasets, and evaluation protocols, making cross-study comparison difficult.
• Current models often generate fluent but factually incorrect or hallucinated content, especially when relying solely on neural sequence models without external knowledge.
• Only a few works consider the coherence and structure of the final related work section, focusing instead on individual citation sentences.
• The field remains largely confined to computational linguistics, with minimal exploration of related work generation in natural sciences or social sciences.