[Paper Review] SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval

Thibault Formal, Carlos Lassance+2 more|arXiv (Cornell University)|Sep 21, 2021

Topic Modeling25 references11 citations

TL;DR

This paper proposes SPLADE v2, an enhanced sparse neural retrieval model that improves effectiveness through max pooling, document-only encoding, and knowledge distillation. It achieves state-of-the-art performance on BEIR with over 9% NDCG@10 gain on TREC DL 2019 and outperforms dense models on zero-shot evaluation.

ABSTRACT

In neural Information Retrieval (IR), ongoing research is directed towards improving the first retriever in ranking pipelines. Learning dense embeddings to conduct retrieval using efficient approximate nearest neighbors methods has proven to work well. Meanwhile, there has been a growing interest in learning \emph{sparse} representations for documents and queries, that could inherit from the desirable properties of bag-of-words models such as the exact matching of terms and the efficiency of inverted indexes. Introduced recently, the SPLADE model provides highly sparse representations and competitive results with respect to state-of-the-art dense and sparse approaches. In this paper, we build on SPLADE and propose several significant improvements in terms of effectiveness and/or efficiency. More specifically, we modify the pooling mechanism, benchmark a model solely based on document expansion, and introduce models trained with distillation. We also report results on the BEIR benchmark. Overall, SPLADE is considerably improved with more than $9$\% gains on NDCG@10 on TREC DL 2019, leading to state-of-the-art results on the BEIR benchmark.

Motivation & Objective

• Address the vocabulary mismatch problem in sparse retrieval by enhancing term-level modeling and expansion mechanisms.
• Improve the effectiveness of sparse neural retrieval models while maintaining efficiency and compatibility with inverted indexes.
• Explore efficient inference alternatives by training a document-only encoder that pre-computes representations.
• Leverage knowledge distillation to boost performance without increasing online inference cost.
• Achieve state-of-the-art results on BEIR and competitive performance on MS MARCO and TREC DL 2019 benchmarks.

Proposed method

• Replace the original SPLADE pooling mechanism with max pooling to better capture salient term representations and improve signal strength.
• Introduce a document-only encoder (SPLADE-doc) that generates sparse document representations without query encoding, enabling full offline pre-computation.
• Apply knowledge distillation to transfer knowledge from a stronger teacher model (e.g., dense or large sparse model) to a smaller student model, improving performance with minimal FLOPS overhead.
• Train models with L1 regularization to enforce sparsity and maintain compatibility with efficient inverted index systems.
• Use a two-stage training strategy: first pre-train the model on a large corpus, then fine-tune on labeled IR datasets using contrastive learning objectives.
• Optimize hyperparameters such as regularization strength (λ) and pooling temperature to balance sparsity and performance.

Experimental results

Research questions

• RQ1 Does replacing the original pooling mechanism in SPLADE with max pooling significantly improve retrieval effectiveness?
• RQ2 Can a document-only encoder model achieve competitive performance while enabling faster, fully pre-computable retrieval?
• RQ3 To what extent does knowledge distillation enhance the performance of sparse neural retrieval models without degrading inference efficiency?
• RQ4 How does SPLADE v2 compare to state-of-the-art dense and sparse models on zero-shot and few-shot evaluation benchmarks like BEIR?
• RQ5 What is the trade-off between model sparsity (average non-zero dimensions per document) and retrieval effectiveness in the proposed models?

Key findings

• SPLADE- max achieves a 2-point improvement in MRR@10 and NDCG@10 over the original SPLADE on MS MARCO, outperforming COIL and DeepImpact.
• The SPLADE-doc model achieves MRR@10 of 29.6% with only 19 non-zero weights per document on MS MARCO, matching performance of doc2query-T5 while being more efficient.
• DistilSPLADE- max achieves SOTA results on the BEIR benchmark, outperforming all baselines on 11 out of 14 datasets and achieving an average NDCG@10 of 0.506 on zero-shot evaluation.
• On TREC DL 2019, SPLADE v2 achieves over 9% gain in NDCG@10 compared to the original SPLADE, establishing a new state-of-the-art.
• The distilled model maintains high performance even at low FLOPS (e.g., 0.35 MRR@10 at ~0.3 FLOPS), demonstrating strong efficiency-accuracy trade-offs.
• The effectiveness of max pooling is consistent across multiple datasets, with SPLADE- max showing superior performance in both effectiveness and efficiency (as shown in Figure 1).