[Paper Review] GOAL: Towards Benchmarking Few-Shot Sports Game Summarization

Jiaan Wang, Tingyi Zhang+1 more|arXiv (Cornell University)|Jul 18, 2022

Video Analysis and Summarization4 citations

TL;DR

This paper introduces GOAL, the first English-language dataset for few-shot sports game summarization, comprising 103 commentary-news pairs and 2,160 unlabeled commentaries. Despite strong abstractive baselines like LED achieving ROUGE-L scores of 24.3 on test, the results reveal persistent challenges due to limited data and stylistic divergence between commentaries and news, highlighting the need for better modeling of sports-specific language and knowledge integration.

ABSTRACT

Sports game summarization aims to generate sports news based on real-time commentaries. The task has attracted wide research attention but is still under-explored probably due to the lack of corresponding English datasets. Therefore, in this paper, we release GOAL, the first English sports game summarization dataset. Specifically, there are 103 commentary-news pairs in GOAL, where the average lengths of commentaries and news are 2724.9 and 476.3 words, respectively. Moreover, to support the research in the semi-supervised setting, GOAL additionally provides 2,160 unlabeled commentary documents. Based on our GOAL, we build and evaluate several baselines, including extractive and abstractive baselines. The experimental results show the challenges of this task still remain. We hope our work could promote the research of sports game summarization. The dataset has been released at https://github.com/krystalan/goal.

Motivation & Objective

• To address the lack of English-language datasets for sports game summarization, which has limited global research in the field.
• To create a benchmark dataset for few-shot sports game summarization using real-time English commentaries and corresponding news articles.
• To support semi-supervised learning by providing 2,160 additional unlabeled commentary documents.
• To evaluate the difficulty of the task through extractive and abstractive baselines, revealing persistent challenges in style transfer and factual consistency.
• To promote future research in multi-lingual, knowledge-enhanced, and graph-based modeling for sports summarization.

Proposed method

• The authors collected 2,263 football game commentaries from Goal.com between 2016 and 2020, of which 103 were paired with official sports news.
• The dataset is split 63/20/20 for training, validation, and testing, with 2,160 additional unlabeled commentaries provided for semi-supervised learning.
• They evaluated both extractive baselines (Longest, TextRank, PacSum) and abstractive models (PGN, LED) using ROUGE-1/2/L as automatic metrics.
• The LED model was fine-tuned with a learning rate of 3e-5, batch size 4, and 10 epochs, using truncation to 4096 input and 1024 output tokens.
• To analyze model behavior, key verbs in generated news were examined to assess understanding of sports-specific events.
• The study leverages the Hugging Face LED-base-16384 model with sparse attention and beam search decoding during inference.

Experimental results

Research questions

• RQ1 How does the performance of extractive and abstractive models compare on a low-resource, few-shot English sports summarization benchmark?
• RQ2 To what extent do current models capture key events and maintain factual consistency in sports news generation?
• RQ3 Can models trained on limited English data effectively learn the stylistic and semantic shift from colloquial commentaries to formal news?
• RQ4 How effective is semi-supervised learning using the 2,160 unlabeled commentaries in improving summarization performance?
• RQ5 What role does external knowledge play in mitigating hallucinations and improving understanding of complex sports events?

Key findings

• The LED abstractive model achieved the highest ROUGE-L score of 24.3 on the test set, outperforming extractive methods and PGN, indicating the advantage of abstractive generation in this task.
• Extractive baselines such as Longest and TextRank achieved ROUGE-L scores below 20, indicating poor performance due to stylistic divergence between commentaries and news.
• The PGN model achieved a ROUGE-L of 21.4 on test, showing improved performance over extractive methods but still limited by long-context processing inefficiency.
• Manual analysis revealed that the LED model frequently repeated phrases and failed to capture complex or uncommon events, such as 'blocking', suggesting poor generalization to rare sports actions.
• The model correctly predicted simple verbs like 'beat' but struggled with nuanced or complex actions, indicating limited understanding of sports-specific language.
• The study confirms that the few-shot setting with only 103 labeled samples severely limits model learning, highlighting the need for external knowledge or multi-task learning.