[論文レビュー] Overview of the SciHigh Track at FIRE 2025: Research Highlight Generation from Scientific Papers

Tohida Rehman, Debarshi Kumar Sanyal+1 more|arXiv (Cornell University)|Jan 1, 2026

Biomedical Text Mining and Ontologies被引用数 0

ひとこと要約

The paper reports the SciHigh track at FIRE 2025, where teams generated bullet-point highlights from abstracts using the MixSub dataset; the best ROUGE-L F1 score was 23.45% for a fine-tuned Pegasus model.

ABSTRACT

`SciHigh: Research Highlight Generation from Scientific Papers' focuses on the task of automatically generating concise, informative, and meaningful bullet-point highlights directly from scientific abstracts. The goal of this task is to evaluate how effectively computational models can generate highlights that capture the key contributions, findings, and novelty of a paper in a concise form. Highlights help readers grasp essential ideas quickly and are often easier to read and understand than longer paragraphs, especially on mobile devices. The track uses the MixSub dataset \cite{10172215}, which provides pairs of abstracts and corresponding author-written highlights. In this inaugural edition of the track, 12 teams participated, exploring various approaches, including pre-trained language models, to generate highlights from this scientific dataset. All submissions were evaluated using established metrics such as ROUGE, METEOR, and BERTScore to measure both alignment with author-written highlights and overall informativeness. Teams were ranked based on ROUGE-L scores. The findings suggest that automatically generated highlights can reduce reading effort, accelerate literature reviews, and enhance metadata for digital libraries and academic search platforms. SciHigh provides a dedicated benchmark for advancing methods aimed at concise and accurate highlight generation from scientific writing.

研究の動機と目的

• Enable quick comprehension of scientific contributions through bullet-point highlights.
• Provide a benchmark for automatic highlight generation from abstracts.
• Evaluate transformer- and retrieval-based approaches on a multidisciplinary dataset.
• Assess how well generated highlights align with author-written highlights and inform readers.

提案手法

• Use the MixSub dataset with abstract–highlight pairs across multiple domains.
• Fine-tune pre-trained language models (e.g., Pegasus, T5, BART) for highlight generation.
• Evaluate submissions with ROUGE-1, ROUGE-2, and ROUGE-L (ranking by ROUGE-L F1).
• Provide a held-out test set with masked ground-truth highlights and release evaluation code for reproducibility.
• Explore hybrid extractive–abstractive and retrieval-augmented strategies.

実験結果

リサーチクエスチョン

• RQ1 Can automatic models generate concise, author-like research highlights from abstracts?
• RQ2 How do different model families (extractive, abstractive, hybrid) perform on cross-domain scientific text?
• RQ3 What is the best-performing approach under ROUGE-L F1 for SciHigh on MixSub?
• RQ4 What is the level of alignment between generated highlights and author-written highlights across domains?

主な発見

Group Name	Run Submission (Best Run)	ROUGE-L F1	Rank
Text_highlights_gen	run1	23.45	1
AiNauts	run1	23.24	2
SVNIT_CSE	run1	23.02	3
NLPFusion	run2	22.96	4
The NLP Explorers	run2	22.94	5
NIT_PATNA_2025	run1	22.42	6
MUCS	run1	22.08	7
JU_CSE_PR_KS	run1	22.06	8
SCaLAR	run1	20.33	9
Ayanika	run1	17.91	10

• Twelve teams participated; best ROUGE-L F1 was 23.45% (Text_highlights_gen with Pegasus-based model).
• Second best ROUGE-L F1 was 23.24% (AiNauts with extractive–abstractive method).
• Third to tenth place scores ranged from 23.02% to 17.91% across teams and models.
• Teams employed a mix of fine-tuned transformers, extractive ranking, and retrieval-augmented methods.
• Evaluation used ROUGE-1, ROUGE-2, and ROUGE-L with final ranking based on ROUGE-L F1.
• The track establishes a benchmark and demonstrates that automatic highlights can aid literature review and metadata quality, while highlighting remaining challenges such as cross-sentence information integration.