[Paper Review] Quantified Reproducibility Assessment of NLP Results

Anja Belz, Maja Popović+1 more|arXiv (Cornell University)|Apr 12, 2022

Software Engineering Research4 citations

TL;DR

This paper proposes Quantified Reproducibility Assessment (QRA), a metrology-based method that produces a single, comparable score estimating the reproducibility of NLP system-evaluation measure combinations using multiple reproductions. It enables objective, scale-invariant comparison across diverse studies and identifies design factors—like feature types or evaluation types—that affect reproducibility, with random forest models showing best reproducibility and logistic regressors with domain-specific features showing worst.

ABSTRACT

This paper describes and tests a method for carrying out quantified reproducibility assessment (QRA) that is based on concepts and definitions from metrology. QRA produces a single score estimating the degree of reproducibility of a given system and evaluation measure, on the basis of the scores from, and differences between, different reproductions. We test QRA on 18 system and evaluation measure combinations (involving diverse NLP tasks and types of evaluation), for each of which we have the original results and one to seven reproduction results. The proposed QRA method produces degree-of-reproducibility scores that are comparable across multiple reproductions not only of the same, but of different original studies. We find that the proposed method facilitates insights into causes of variation between reproductions, and allows conclusions to be drawn about what changes to system and/or evaluation design might lead to improved reproducibility.

Motivation & Objective

• To address the lack of standardized, objective, and comparable methods for assessing reproducibility in NLP research.
• To develop a quantifiable, scale-invariant metric that captures reproducibility across different NLP systems and evaluation measures.
• To enable insights into which system and evaluation design choices contribute to variation in reproducibility.
• To support both post-hoc assessment of existing studies and future integration of reproducibility testing into method development.

Proposed method

• QRA is grounded in metrology principles, using the concept of 'degree of reproducibility' as a normalized score (CV∗) derived from the coefficient of variation of multiple reproduction results.
• The method computes CV∗ as the ratio of the standard deviation of reproduction scores to their mean, ensuring scale invariance.
• It incorporates differences in system and evaluation design by treating them as conditions of measurement, allowing for formal comparison of reproducibility across varying experimental setups.
• QRA enables both post-hoc assessment of existing studies and prospective repeatability testing during method development.
• The approach uses a standardized set of measurement conditions to assess how changes in system or evaluation design affect reproducibility.
• It supports comparative analysis across different original studies by normalizing scores and accounting for design (dis)similarities.

Experimental results

Research questions

• RQ1 How can reproducibility in NLP be assessed in a way that is objective, comparable across studies, and independent of scale?
• RQ2 What role do differences in system architecture and evaluation design play in observed variation between reproductions?
• RQ3 Which NLP system and evaluation measure combinations exhibit the highest or lowest reproducibility, and why?
• RQ4 Can QRA identify specific design choices that lead to improved or degraded reproducibility?

Key findings

• The QRA method produces a single, scale-invariant score (CV∗) that enables direct comparison of reproducibility across different NLP systems and evaluation measures.
• Random forest models using syntactic features showed the best reproducibility, while logistic regressors using domain-specific features showed the worst.
• Human-assessed evaluations were not uniformly less reproducible; for example, Stance Identifiability in the PASS system was among the most reproducible.
• Metric-assessed wF1 scores for mult-base, mult-dom, and mult-emb systems were among the least reproducible, indicating potential issues with automatic metrics.
• The method successfully identified that feature choice was a key factor associated with variation in reproducibility for essay scoring systems.
• Post-hoc QRA revealed that persistent high CV∗ values in human evaluations may indicate fundamental flaws in the evaluation method itself.