[Paper Review] Semantic Document Distance Measures and Unsupervised Document Revision Detection

Xiaofeng Zhu, Diego Klabjan+1 more|arXiv (Cornell University)|Sep 5, 2017

Web Data Mining and Analysis7 references3 citations

TL;DR

This paper proposes two novel semantic document distance measures, word vector-based Dynamic Time Warping (wDTW) and word vector-based Tree Edit Distance (wTED), to improve unsupervised document revision detection. By leveraging word2vec embeddings and modeling revision detection as a minimum cost branching problem in Apache Spark, the method achieves higher precision and recall than state-of-the-art approaches like Word Mover's Distance (WMD), with significantly reduced computation time on large-scale corpora.

ABSTRACT

In this paper, we model the document revision detection problem as a minimum cost branching problem that relies on computing document distances. Furthermore, we propose two new document distance measures, word vector-based Dynamic Time Warping (wDTW) and word vector-based Tree Edit Distance (wTED). Our revision detection system is designed for a large scale corpus and implemented in Apache Spark. We demonstrate that our system can more precisely detect revisions than state-of-the-art methods by utilizing the Wikipedia revision dumps https://snap.stanford.edu/data/wiki-meta.html and simulated data sets.

Motivation & Objective

• To address the limitations of traditional document similarity measures—such as fingerprinting, Levenshtein distance, and VSM—in capturing semantic meaning and scalability.
• To develop scalable, semantic-aware document distance measures that outperform existing methods like WMD in both accuracy and efficiency for large-scale revision detection.
• To model document revision detection as a minimum cost branching problem using document distance scores as arc weights, enabling efficient network-based revision relationship inference.
• To implement and evaluate the system on Wikipedia revision dumps and simulated datasets, demonstrating robustness and scalability in real-world settings.

Proposed method

• Represent words using pre-trained word2vec embeddings to capture semantic meaning, replacing static word representations.
• Compute document distances by comparing paragraph-level representations using Dynamic Time Warping (DTW) and Tree Edit Distance (TED), with word vector similarity as the base distance metric.
• Model document revision detection as a minimum cost branching problem in a directed graph, where nodes are documents and edge weights are semantic distances.
• Split document distance computation into paragraph-level comparisons and distribute processing using Apache Spark to enable parallelization and scalability.
• Use a novel distance function, DistPara, to compute paragraph-level similarity based on word vector concatenation, improving accuracy over Euclidean distance between paragraph vectors.
• Apply the minimum branching algorithm to infer the optimal revision network, identifying the most likely revision history across a large corpus.

Experimental results

Research questions

• RQ1 Can semantic document distance measures based on word embeddings improve the accuracy of unsupervised document revision detection compared to traditional methods?
• RQ2 How do wDTW and wTED compare in performance and efficiency to Word Mover’s Distance (WMD), VSM, and PV-based methods on large-scale document corpora?
• RQ3 Can the minimum cost branching formulation effectively reconstruct revision networks from document similarity scores without supervision?
• RQ4 Does computing distances at the paragraph level with parallel processing in Apache Spark maintain high accuracy while significantly reducing runtime?
• RQ5 How do wDTW and wTED perform as the corpus size and revision chain length increase?

Key findings

• wDTW achieved the highest F-measure (0.85) on Wikipedia revision dumps and consistently outperformed WMD, VSM, and PV-based methods in precision and recall.
• wTED delivered strong performance with 14% lower runtime than wDTW, making it a more efficient alternative while maintaining high accuracy.
• On simulated datasets, wDTW maintained stable performance as corpus size increased, whereas WMD and VSM showed faster performance degradation.
• wDTW and wTED reduced running time by over 90% compared to WMD, which required 515 hours on Wikipedia dumps, while wDTW completed in 13 hours 27 minutes.
• The DistPara distance function improved accuracy over Euclidean distance between paragraph vectors, as shown by better F-measure scores in comparative experiments.
• The system scaled effectively with Apache Spark, reducing computation time from days (WMD) to hours (wDTW) on large corpora.