[Paper Review] Document-Level Definition Detection in Scholarly Documents: Existing Models, Error Analyses, and Future Directions

Dongyeop Kang, Andrew Head+4 more|arXiv (Cornell University)|Oct 11, 2020

Topic Modeling27 references4 citations

TL;DR

This paper proposes HEDDE x, a new document-level definition detection system that improves upon state-of-the-art models by integrating syntactic features, transformer encoders, and heuristic filters. It achieves a 14.4 F1-point gain over the leading system on document-level evaluation, highlighting the critical need for document-structure-aware modeling to boost recall in scholarly document processing.

ABSTRACT

The task of definition detection is important for scholarly papers, because papers often make use of technical terminology that may be unfamiliar to readers. Despite prior work on definition detection, current approaches are far from being accurate enough to use in real-world applications. In this paper, we first perform in-depth error analysis of the current best performing definition detection system and discover major causes of errors. Based on this analysis, we develop a new definition detection system, HEDDEx, that utilizes syntactic features, transformer encoders, and heuristic filters, and evaluate it on a standard sentence-level benchmark. Because current benchmarks evaluate randomly sampled sentences, we propose an alternative evaluation that assesses every sentence within a document. This allows for evaluating recall in addition to precision. HEDDEx outperforms the leading system on both the sentence-level and the document-level tasks, by 12.7 F1 points and 14.4 F1 points, respectively. We note that performance on the high-recall document-level task is much lower than in the standard evaluation approach, due to the necessity of incorporation of document structure as features. We discuss remaining challenges in document-level definition detection, ideas for improvements, and potential issues for the development of reading aid applications.

Motivation & Objective

• Address the gap in high-recall definition detection for scholarly documents, where current models are evaluated only on randomly sampled sentences.
• Identify and analyze key error sources in the state-of-the-art definition detection model to guide improvements.
• Develop a new system, HEDDE x, that enhances precision and recall by incorporating syntactic features, transformer encoders, and heuristic filters.
• Introduce and evaluate a new document-level definition detection benchmark that assesses all sentences in a document, enabling full-recall evaluation.
• Explore challenges in detecting mathematical symbols, distinguishing definitions from descriptions, and enabling practical reading aid applications.

Proposed method

• Perform in-depth error analysis on the current state-of-the-art model to identify failure modes, including misclassification of mathematical symbols and confusion between definitions and descriptions.
• Design HEDDE x by extending a state-of-the-art model with syntactic features (e.g., dependency parsing), BERT-based transformer encoders, and rule-based heuristic filters to refine predictions.
• Evaluate HEDDE x on a standard sentence-level benchmark and introduce a new document-level evaluation protocol that assesses every sentence in a document to measure recall.
• Leverage document-level features such as section structure and positional context to improve detection of first-time term introductions.
• Apply data augmentation and domain-specific fine-tuning using models like SciBERT to improve coverage of rare or technical terms.
• Use annotated datasets including labeled mathematical definitions from Vanetik et al. (2020) to enhance training data for symbol recognition.

Experimental results

Research questions

• RQ1 What are the primary causes of failure in current state-of-the-art definition detection models when applied to scholarly documents?
• RQ2 How does document-level evaluation—assessing all sentences in a document—reveal limitations not exposed by standard sentence-level sampling?
• RQ3 To what extent can syntactic features, transformer encoders, and heuristic filters improve definition detection performance on both sentence-level and document-level tasks?
• RQ4 What challenges remain in detecting mathematical symbols and distinguishing definitions from descriptive text in scholarly papers?
• RQ5 How can definition detection systems be adapted to support real-world reading aid applications with high precision and recall?

Key findings

• HEDDE x outperforms the leading system by 12.7 F1 points on the standard sentence-level benchmark, demonstrating significant gains in precision and recall.
• On the proposed document-level evaluation, HEDDE x achieves a 14.4 F1-point improvement over the state of the art, underscoring the importance of document-structure-aware modeling.
• Error analysis reveals that mathematical symbol detection is a major weakness, with low coverage in training data leading to poor generalization on symbols like $s^{task}$.
• Distinguishing definitions from descriptions remains a key challenge, as both share similar surface patterns but differ fundamentally in meaning and structure.
• Document-level features such as section position and term introduction patterns significantly improve detection, especially for first-time term occurrences.
• The study identifies the need for better data coverage, including annotated mathematical definitions, and for improved categorization of definition types (e.g., formulaic, acronymic) in future systems.