[Paper Review] A Content-Based Novelty Measure for Scholarly Publications: A Proof of Concept

Haining Wang|arXiv (Cornell University)|Jan 8, 2024

scientometrics and bibliometrics research4 citations

TL;DR

This paper proposes a content-based novelty measure for scholarly publications using surprisal from a language model trained on Wikipedia, quantifying novelty as the 'surprise' of word sequences relative to scholarly discourse. It demonstrates statistically significant higher surprisal in papers judged novel by experts (p = 0.005), establishing construct validity and offering an interpretable, open-source tool for assessing novelty at scale.

ABSTRACT

Model Architecture GPT-2 124M Variant Number of Layers (n_layer): 12 Number of Attention Heads (n_head): 12 Embedding Size (n_embd): 768 Total Parameters: Approximately 124 million Dropout Rate: 0.0 (no dropout applied during training) Bias in LayerNorm and Linear Layers: Not used (bias = False) Refer to Radford et al. (2019) for details. Training Scheme Dataset The model utilizes the wikipedia_en dataset, which consists of English Wikipedia articles, with data up to the cutoff date of March 1, 2022, sourced from the Hugging Face datasets library (datasets.load_dataset("wikipedia", "20220301.en")). The version of datasets being used is 2.9.0. Tokenizer The specific encoding function used is tiktoken.get_encoding("gpt2"), which retrieves the tokenizer specific to GPT-2 and includes a vocabulary of 50,304 tokens. An end-of-text token (EOT), denoted by enc.eot_token (for example, 50,256 for GPT-2 BPE), is appended to each sequence of token IDs. The version of tiktoken being used is 0.2.0. Batch and Block Configuration Batch Size: 16 (micro-batch size) Block Size: 1,024 tokens Gradient Accumulation Steps: 5 Effective Total Batch Size: Approximately 327,680 tokens (16 * 1024 * 5 * 4 A100s) Optimizer (AdamW) Initial Learning Rate: 6e-4 Final Minimum Learning Rate: 6e-5 Weight Decay: 1e-1 Beta1: 0.9 Beta2: 0.95 Gradient Clipping Value: 1.0 Epsilon (eps): 1e-5 Learning Rate Decay Warmup Iterations: 2,000 Decay Iterations: 141,000 (aligned with max_iters) Training Iterations Maximum Iterations (max_iters): 141,000 Approximate Number of Epochs: 10 (totaling around 46 billion tokens) Evaluation Scheme Evaluation Interval: Every 1,000 iterations Number of Evaluation Iterations: 200 Other Dependencies For further details on other dependencies, refer to requirements.txt at https://github.com/Wang-Haining/noveval. References Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 9.

Motivation & Objective

• To develop an automated, interpretable measure of scholarly novelty that operates at the content level rather than relying on citations or keywords.
• To validate the measure using face validity (alignment with scientific common sense) and construct validity (correlation with expert judgments).
• To enable fine-grained, section-level novelty assessment in scholarly manuscripts using information-theoretic principles.
• To provide a reproducible, open-source tool for editors, reviewers, and policymakers to evaluate novelty in academic outputs.
• To explore the relationship between novelty, creativity, and interdisciplinarity in scientific research.

Proposed method

• The method computes surprisal using a GPT-2 language model pre-trained on English Wikipedia to estimate the probability of each word in a manuscript's text.
• Surprisal is calculated as the negative log-probability of each word given its preceding context, with average surprisal computed over 1,024-token windows conditioned on at least 256 prior tokens.
• The model is applied to the concatenated abstract and body text of manuscripts, focusing on methodological and findings sections to assess novelty.
• The measure treats novelty as atypical word combinations—deviations from the expected distribution of scholarly language—using information theory.
• Construct validity is tested via the known-groups technique, comparing average surprisal between papers unanimously rated as 'novel' and 'normal' by domain experts.
• The AVA dataset, comprising 80 working notes from authorship verification competitions, is used to evaluate the method on real-world academic texts.

Experimental results

Research questions

• RQ1 Can surprisal from a language model trained on general Wikipedia text reliably measure content-based novelty in scholarly publications?
• RQ2 Does the proposed measure show construct validity by distinguishing between papers judged novel and non-novel by domain experts?
• RQ3 To what extent does the measure reflect scientific common sense and expert intuition about novelty?
• RQ4 How do anomalies in surprisal scores relate to textual artifacts such as mathematical notation, informal syntax, or narrative consistency?
• RQ5 Can the measure be applied at the section level to detect novelty in methodological and findings components of research papers?

Key findings

• The average surprisal of papers unanimously rated as novel by domain experts was significantly higher than that of normal papers (p = 0.005), confirming construct validity.
• A one-tailed Welch’s t-test yielded a t-statistic of 2.6, indicating that novel papers had substantially higher surprisal values than normal papers.
• Eight papers with surprisal above 5.0 were deemed less novel by experts; analysis revealed that excessive mathematical notation, informal syntax, and narrative consistency in incremental work contributed to false alarms.
• The measure demonstrated interpretability through its information-theoretic foundation, allowing direct understanding of 'surprise' as a function of word predictability.
• The method is reproducible and publicly available at github.com/Wang-Haining/noveval under a permissive license.
• The study establishes a proof-of-concept for content-based novelty measurement that avoids reliance on citations, keywords, or coarse-grained knowledge units.