[Paper Review] Large scale citation matching using Apache Hadoop

Mateusz Fedoryszak, Dominika Tkaczyk+1 more|arXiv (Cornell University)|Mar 26, 2013

Semantic Web and Ontologies18 references4 citations

TL;DR

This paper presents a scalable citation matching system built on Apache Hadoop that uses approximate indexing and MapReduce to efficiently match citation strings to their referenced publications across large-scale scholarly datasets. By leveraging token-based similarity, trigram and token matching, and a heuristic filtering approach with a custom index, the system achieves high performance on a 12M-citation dataset, reducing matching time to under 3 hours on a 4-node cluster.

ABSTRACT

During the process of citation matching links from bibliography entries to referenced publications are created. Such links are indicators of topical similarity between linked texts, are used in assessing the impact of the referenced document and improve navigation in the user interfaces of digital libraries. In this paper we present a citation matching method and show how to scale it up to handle great amounts of data using appropriate indexing and a MapReduce paradigm in the Hadoop environment.

Motivation & Objective

• To address the challenge of scaling citation matching to large scholarly datasets with millions of citations.
• To design a scalable, distributed solution using Apache Hadoop and the MapReduce paradigm for efficient citation resolution.
• To develop an approximate indexing strategy that accelerates heuristic matching without sacrificing accuracy.
• To enable large-scale citation matching for digital libraries, scientometric analysis, and knowledge graph construction.

Proposed method

• Citation parsing using Conditional Random Fields (CRF) with 42 features, including character class, punctuation, dictionary lookup, and context from neighboring tokens.
• Computation of trigram and token similarity between metadata fields (author, title, journal, year) using Jaccard-based similarity measures.
• Heuristic matching via an approximate index built using token rotations and MapReduce, enabling fast retrieval of candidate documents per citation.
• Use of a MapFile structure in Hadoop to store and efficiently query the index, with sorted keys and fast random access.
• Implementation of a two-phase MapReduce pipeline: (1) citation extraction and candidate retrieval, (2) similarity scoring and best-match selection using SVM and string metrics.
• Index construction via map tasks (token extraction and rotation generation) and reduce tasks (grouping by token and persisting to SequenceFile).

Experimental results

Research questions

• RQ1 How can citation matching be scaled to handle millions of citations efficiently in a distributed environment?
• RQ2 What indexing strategy enables fast, approximate matching of citation strings while preserving accuracy?
• RQ3 How can metadata parsing and similarity computation be optimized for noisy, heterogeneous citation strings?
• RQ4 What performance gains can be achieved using Hadoop’s MapReduce model for citation matching workloads?

Key findings

• The system successfully processed over 12 million citations from the PMC Open Access Subset using a 4-node Hadoop cluster.
• Index building completed in 57 seconds, with 13 map and 2 reduce tasks, demonstrating efficient parallel construction.
• Heuristic matching phase took 3 hours and 1 second across 745 map tasks, significantly reducing the search space.
• Best-match selection took 2 hours and 51 minutes using 996 map tasks and 1 reduce task, ensuring high-precision results.
• The overall matching pipeline completed in under 3 hours, proving the feasibility of large-scale citation matching using Hadoop.
• The use of token rotations and approximate indexing enabled fast candidate retrieval while maintaining high recall for matching citations.