[Paper Review] Synthetic vs. Real Reference Strings for Citation Parsing, and the Importance of Re-training and Out-Of-Sample Data for Meaningful Evaluations: Experiments with GROBID, GIANT and Cora

Mark Grennan, Joeran Beel|arXiv (Cornell University)|Apr 22, 2020

Natural Language Processing Techniques31 references4 citations

TL;DR

This study evaluates synthetic versus real reference strings for training citation parsers, demonstrating that synthetic data from the GIANT dataset performs as well as real data when training GROBID (F1 = 0.74). Retraining significantly boosts performance (+30% F1), and including diverse field types during training improves results even if not evaluated on them (+13.5% F1), proving synthetic data is viable for citation parsing models.

ABSTRACT

Citation parsing, particularly with deep neural networks, suffers from a lack of training data as available datasets typically contain only a few thousand training instances. Manually labelling citation strings is very time-consuming, hence synthetically created training data could be a solution. However, as of now, it is unknown if synthetically created reference-strings are suitable to train machine learning algorithms for citation parsing. To find out, we train Grobid, which uses Conditional Random Fields, with a) human-labelled reference strings from 'real' bibliographies and b) synthetically created reference strings from the GIANT dataset. We find that both synthetic and organic reference strings are equally suited for training Grobid (F1 = 0.74). We additionally find that retraining Grobid has a notable impact on its performance, for both synthetic and real data (+30% in F1). Having as many types of labelled fields as possible during training also improves effectiveness, even if these fields are not available in the evaluation data (+13.5% F1). We conclude that synthetic data is suitable for training (deep) citation parsing models. We further suggest that in future evaluations of reference parsers both evaluation data similar and dissimilar to the training data should be used for more meaningful evaluations.

Motivation & Objective

• To assess whether synthetic reference strings from the GIANT dataset are suitable for training citation parsing models.
• To compare the performance of GROBID when trained on synthetic versus real human-labeled citation strings.
• To investigate the impact of retraining and inclusion of diverse field types during training on model effectiveness.
• To evaluate the importance of out-of-sample data in meaningful performance assessment of citation parsers.
• To provide evidence-based recommendations for training and evaluating citation parsing systems.

Proposed method

• Trained GROBID, a CRF-based citation parser, on both synthetic reference strings from the GIANT dataset and real human-labeled citation strings from the Cora dataset.
• Applied retraining to assess its impact on F1 score improvements across both synthetic and real data.
• Incorporated a broader range of labeled fields during training, even when those fields were not present in evaluation data.
• Used out-of-sample data—data not seen during training—to evaluate model generalization and robustness.
• Measured performance using F1 score as the primary metric across multiple experimental configurations.
• Compared results across training data types (synthetic vs. real), retraining conditions, and field diversity to isolate key performance factors.

Experimental results

Research questions

• RQ1 Can synthetic reference strings from the GIANT dataset effectively train citation parsing models to match performance on real human-labeled data?
• RQ2 How does retraining affect the F1 performance of citation parsers like GROBID when trained on synthetic or real data?
• RQ3 Does including a wider variety of labeled fields during training improve parser performance, even if those fields are not evaluated?
• RQ4 To what extent does out-of-sample data improve the validity of citation parser evaluations?
• RQ5 Is synthetic data a viable alternative to real data for training and evaluating citation parsing systems?

Key findings

• Synthetic reference strings from the GIANT dataset produced comparable F1 scores (0.74) to real human-labeled strings when training GROBID.
• Retraining the model led to a 30% increase in F1 score on both synthetic and real training data, highlighting its significant impact.
• Including diverse field types during training improved F1 by 13.5%, even when those fields were not present in the evaluation data.
• The study demonstrates that synthetic data is a viable and effective alternative to real data for training citation parsing models.
• Using both similar and dissimilar out-of-sample data in evaluation leads to more meaningful and robust performance assessments.
• The results support the use of synthetic data in training citation parsers, provided retraining and diverse field representation are applied.