[Paper Review] Content-Based Citation Recommendation

Chandra Bhagavatula, Sergey Feldman+2 more|arXiv (Cornell University)|Feb 22, 2018

Topic Modeling20 references12 citations

TL;DR

This paper proposes a content-based citation recommendation system that uses neural network embeddings of document text to recommend relevant citations without relying on metadata like authors or venues. By first selecting nearest neighbors in a learned vector space and then reranking them with a discriminative model, the method achieves state-of-the-art performance on PubMed and DBLP, outperforming prior work by over 18% in F1@20 and 22% in MRR, even when metadata are missing.

ABSTRACT

We present a content-based method for recommending citations in an academic paper draft. We embed a given query document into a vector space, then use its nearest neighbors as candidates, and rerank the candidates using a discriminative model trained to distinguish between observed and unobserved citations. Unlike previous work, our method does not require metadata such as author names which can be missing, e.g., during the peer review process. Without using metadata, our method outperforms the best reported results on PubMed and DBLP datasets with relative improvements of over 18% in F1@20 and over 22% in MRR. We show empirically that, although adding metadata improves the performance on standard metrics, it favors self-citations which are less useful in a citation recommendation setup. We release an online portal (http://labs.semanticscholar.org/citeomatic/) for citation recommendation based on our method, and a new dataset OpenCorpus of 7 million research articles to facilitate future research on this task.

Motivation & Objective

• To address the challenge of citation recommendation in scenarios where metadata (e.g., author names, venues) are missing, such as during peer review or early research stages.
• To develop a scalable, content-only citation recommendation method that does not require retraining when new documents are added to the corpus.
• To improve upon existing citation recommendation systems by eliminating reliance on metadata, which can introduce bias toward self-citations.
• To release a large-scale, publicly accessible dataset and web-based tool to support future research in citation recommendation.

Proposed method

• The method embeds all documents, including the query, into a shared vector space using a neural network trained on textual content (title and abstract) only.
• Candidate citations are selected via k-nearest neighbors (KNN) search in the vector space, based on semantic similarity between the query and candidate documents.
• A separate discriminative neural network is trained to distinguish between observed citations (positive pairs) and unobserved citation pairs (negative pairs), using the query-document pair as input.
• The model is trained via batched stochastic gradient descent, enabling efficient scaling to large corpora like OpenCorpus.
• The system is designed to be non-transductive: new documents can be embedded into the existing vector space without retraining the entire model.
• The final recommendation list is generated by reranking candidates using the output probability of the discriminative model.

Experimental results

Research questions

• RQ1 Can a citation recommendation system achieve state-of-the-art performance using only the textual content of documents, without relying on metadata such as authors or publication venues?
• RQ2 How does the performance of a content-only citation recommendation system compare to metadata-informed baselines, especially in terms of fairness and avoidance of self-citation bias?
• RQ3 To what extent does adding metadata improve performance, and does this improvement come at the cost of introducing undesirable citation biases?
• RQ4 Can a neural network-based approach scale efficiently to large-scale citation recommendation tasks, such as those involving millions of documents?
• RQ5 How does the proposed method perform on large, realistic datasets compared to previous systems that use pruned or small test sets?

Key findings

• The proposed content-based method outperforms the best reported results on PubMed and DBLP by over 18% in F1@20 and over 22% in MRR, even without using any metadata.
• Adding metadata improves performance on standard metrics but introduces a significant bias toward self-citations, which are less useful in citation recommendation.
• The method remains robust and effective even when metadata are missing, making it suitable for early-stage research or peer review scenarios.
• The system achieves state-of-the-art results on two benchmark datasets using only text embeddings, demonstrating the effectiveness of content-based representation learning.
• The release of OpenCorpus, a dataset of 7 million research articles, addresses limitations in prior datasets, such as small size and heavy pruning.
• The online citation recommendation portal, citeomatic, is publicly available and enables real-time citation suggestions based on text alone.