[Paper Review] Document Embeddings vs. Keyphrases vs. Terms: An Online Evaluation in Digital Library Recommender Systems

Andrew Collins, Joeran Beel|arXiv (Cornell University)|May 27, 2019

Advanced Text Analysis Techniques4 references4 citations

TL;DR

This paper evaluates term-based, document embedding, and keyphrase-based recommendation methods in digital library systems using real-world online user interactions. In a 19-month deployment via Mr. DLib, keyphrases outperformed terms and embeddings in Jabref (0.03% click-through rate), while terms were best in Sowiport (0.1%), with performance differences exceeding 400% between top and bottom methods in each system.

ABSTRACT

Many recommendation algorithms are available to digital library recommender system operators. The effectiveness of algorithms is largely unreported by way of online evaluation. We compare a standard term-based recommendation approach to two promising approaches for related-article recommendation in digital libraries: document embeddings, and keyphrases. We evaluate the consistency of their performance across multiple scenarios. Through our recommender-as-a-service Mr. DLib, we delivered 33.5M recommendations to users of Sowiport and Jabref over the course of 19 months, from March 2017 to October 2018. The effectiveness of the algorithms differs significantly between Sowiport and Jabref (Wilcoxon rank-sum test; p < 0.05). There is a ~400% difference in effectiveness between the best and worst algorithm in both scenarios separately. The best performing algorithm in Sowiport (terms) is the worst performing in Jabref. The best performing algorithm in Jabref (keyphrases) is 70% worse in Sowiport, than Sowiport`s best algorithm (click-through rate; 0.1% terms, 0.03% keyphrases).

Motivation & Objective

• To evaluate the real-world effectiveness of term-based, document embedding, and keyphrase-based recommendation methods in digital library environments.
• To assess whether performance differences between recommendation algorithms are consistent across different digital library platforms.
• To investigate the impact of algorithm choice on user engagement through online click-through rates in production systems.
• To provide empirical evidence on the relative effectiveness of modern representation learning techniques (embeddings, keyphrases) versus traditional term-based methods in real-world digital library recommender systems.

Proposed method

• Deployed a production-grade recommender-as-a-service (Mr. DLib) to deliver recommendations to users of Sowiport and Jabref over 19 months (March 2017 – October 2018).
• Generated recommendations using three distinct representation methods: TF-IDF terms, dense document embeddings (e.g., from neural networks), and keyphrases extracted from metadata and text.
• Measured user engagement via click-through rates (CTR) on recommended articles in real-time, logged through production systems.
• Applied statistical analysis (Wilcoxon rank-sum test) to compare algorithm performance across the two digital library platforms.
• Used online evaluation to assess algorithm effectiveness under real user conditions, avoiding the limitations of offline benchmarks.
• Evaluated performance consistency across two distinct domains: Sowiport (social sciences) and Jabref (computer science and academic software).

Experimental results

Research questions

• RQ1 How do term-based, document embedding-based, and keyphrase-based recommendation algorithms compare in terms of online user engagement in digital libraries?
• RQ2 Does the performance ranking of recommendation algorithms remain consistent across different digital library platforms?
• RQ3 What is the magnitude of performance difference between the best and worst-performing recommendation methods in real-world settings?
• RQ4 Can keyphrases or document embeddings outperform traditional term-based methods in production digital library recommender systems?
• RQ5 How do the click-through rates of different recommendation strategies vary between the Sowiport and Jabref platforms?

Key findings

• The performance of recommendation algorithms varied significantly between Sowiport and Jabref, with no single method being universally best.
• In Sowiport, the term-based method achieved the highest click-through rate (0.1%), while in Jabref, the keyphrase-based method performed best (0.03% CTR).
• The best-performing algorithm in one system was the worst in the other, indicating strong domain dependency.
• The performance gap between the best and worst algorithm was approximately 400% in both Sowiport and Jabref.
• The keyphrase-based method achieved 70% lower click-through rate in Sowiport (0.03%) compared to the best-performing method (0.1%) in that system.
• Statistical analysis confirmed significant performance differences between algorithms (p < 0.05, Wilcoxon rank-sum test), validating the reliability of the results.