[Paper Review] SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval

Yang Bai, Xiaoguang Li+7 more|arXiv (Cornell University)|Oct 2, 2020

Topic ModelingComputer Science16 references59 citations

TL;DR

SparTerm directly learns sparse, term-based representations in the full vocabulary by combining an importance predictor with a gating controller, enabling term-weighting and expansion for improved sparse retrieval on MSMARCO.

ABSTRACT

Term-based sparse representations dominate the first-stage text retrieval in industrial applications, due to its advantage in efficiency, interpretability, and exact term matching. In this paper, we study the problem of transferring the deep knowledge of the pre-trained language model (PLM) to Term-based Sparse representations, aiming to improve the representation capacity of bag-of-words(BoW) method for semantic-level matching, while still keeping its advantages. Specifically, we propose a novel framework SparTerm to directly learn sparse text representations in the full vocabulary space. The proposed SparTerm comprises an importance predictor to predict the importance for each term in the vocabulary, and a gating controller to control the term activation. These two modules cooperatively ensure the sparsity and flexibility of the final text representation, which unifies the term-weighting and expansion in the same framework. Evaluated on MSMARCO dataset, SparTerm significantly outperforms traditional sparse methods and achieves state of the art ranking performance among all the PLM-based sparse models.

Motivation & Objective

• Improve BoW-based first-stage retrieval by integrating deep contextual knowledge from PLMs.
• Develop a direct, end-to-end framework that maps BoW to a sparse vocabulary-space representation.
• Balance term weighting and expansion to enhance semantic matching while preserving interpretability and efficiency.

Proposed method

• Introduce an importance predictor that produces a dense term importance distribution over the full vocabulary.
• Introduce a gating controller that generates a binary sparse activation mask for vocabulary terms.
• Combine F (importance) and G (gating) to form p' = F(p) ⊙ G(p), with sparsity controlled by a threshold λ.
• Enable both literal-term activation and expansion-driven activation to bridge lexical gaps.
• Train end-to-end with a ranking objective using positive/negative passage pairs (q, p+, p-).
• Optionally jointly train expansion gates using a passage–target text corpus with L_exp.

Experimental results

Research questions

• RQ1 Can a direct, sparse, vocabulary-wide representation learned from PLMs outperform traditional sparse methods on first-stage retrieval?
• RQ2 How do the importance predictor and gating controller contribute to term weighting and expansion in sparse representations?
• RQ3 What is the impact of literal-only versus expansion-enhanced gating on retrieval effectiveness?
• RQ4 How does SparTerm compare to DeepCT and Doc2Query-family methods on MSMARCO passage retrieval and document ranking?

Key findings

• SparTerm with expansion-enhanced gating achieves state-of-the-art ranking among sparse models on MSMARCO passage retrieval in terms of MRR@10.
• Literal-only SparTerm outperforms DeepCT in MRR and Recall, showing stronger term weighting capability.
• Expansion-enabled SparTerm improves recall across top ranks, demonstrating the benefit of controlled term expansion.
• Compared with Doc2Query-T5, SparTerm (expansion-enhanced) approaches achieve competitive MRR@10 and higher Recall at several thresholds.
• PLM-based sparse methods (DeepCT, Doc2Query-T5, SparTerm) outperform PLM-free sparse methods, indicating effective transfer of contextual knowledge to sparse representations.
• Analysis shows SparTerm expands semantically related terms (e.g., synonyms, co-occurrences) and assigns smoother, more informative importance distributions.