[Paper Review] SPLADE-v3: New baselines for SPLADE

Carlos Lassance, Hervé Déjean+2 more|arXiv (Cornell University)|Mar 11, 2024

Medical Imaging Techniques and Applications4 citations

TL;DR

SPLADE-v3 introduces a new generation of sparse retrieval models that significantly outperform BM25 and SPLADE++ across diverse benchmarks. By enhancing training with multiple negatives, ensemble-based distillation scores, and a hybrid KL-Divergence and MarginMSE loss, SPLADE-v3 achieves state-of-the-art performance, surpassing even cross-encoder re-rankers on key metrics like MRR@10 and nDCG@10.

ABSTRACT

A companion to the release of the latest version of the SPLADE library. We describe changes to the training structure and present our latest series of models -- SPLADE-v3. We compare this new version to BM25, SPLADE++, as well as re-rankers, and showcase its effectiveness via a meta-analysis over more than 40 query sets. SPLADE-v3 further pushes the limit of SPLADE models: it is statistically significantly more effective than both BM25 and SPLADE++, while comparing well to cross-encoder re-rankers. Specifically, it gets more than 40 MRR@10 on the MS MARCO dev set, and improves by 2% the out-of-domain results on the BEIR benchmark.

Motivation & Objective

• To establish a new, more effective baseline for SPLADE-based sparse retrieval models.
• To improve generalization and effectiveness through refined training procedures without major architectural changes.
• To evaluate SPLADE-v3 across diverse, out-of-domain settings including BEIR, MS MARCO, and LoTTE.
• To compare SPLADE-v3's performance against BM25, SPLADE++, and cross-encoder re-rankers in both in-domain and zero-shot retrieval scenarios.
• To release multiple variants (e.g., DistilBERT, Lexical, Doc) to balance effectiveness and inference efficiency.

Proposed method

• Training with 100 negatives per query—50 from top-50 and 50 from top-1k—using a SPLADE++ model to improve negative mining.
• Generating distillation scores via an ensemble of five cross-encoder models (including MS-MARCO MiniLM and TREC DL 2022 models) for improved label quality.
• Applying a rescored distillation process using affine transformations to match the mean and standard deviation of prior distillation distributions.
• Combining two distillation losses: KL-Divergence (λ=1) and MarginMSE (λ=0.05), optimized via cross-validation to balance precision and recall.
• Initializing training from SPLADE++ SelfDistil instead of CoCondenser or DistilBERT, which empirically improves final performance.
• Using min-max normalization per query on distillation scores via the ranx library to ensure consistent scoring across queries.

Experimental results

Research questions

• RQ1 Can improved training procedures—beyond architectural innovation—significantly enhance SPLADE model effectiveness?
• RQ2 Does using multiple negatives and ensemble-based distillation scores lead to measurable gains in retrieval performance?
• RQ3 How does SPLADE-v3 compare to BM25 and SPLADE++ across a broad set of in-domain and out-of-domain retrieval benchmarks?
• RQ4 Can SPLADE-v3 rival or surpass cross-encoder re-rankers when used as a first-stage retriever with top-50 reranking?
• RQ5 What is the trade-off between effectiveness and inference efficiency across SPLADE-v3 variants (e.g., DistilBERT, Lexical, Doc)?

Key findings

• SPLADE-v3 achieves 40.2 MRR@10 on the MS MARCO dev set, a statistically significant improvement over BM25 and SPLADE++.
• On the BEIR benchmark, SPLADE-v3 improves out-of-domain performance by ↑2% compared to SPLADE++ across 13 datasets, with an average nDCG@10 of 51.7.
• In meta-analysis across 44 query sets, SPLADE-v3 outperforms BM25 in 41 out of 44 sets, with only 3 showing minor, non-significant drops.
• SPLADE-v3 outperforms SPLADE++ SelfDistil in 43 out of 44 query sets, with the only exception being Quora, where performance slightly decreases.
• When re-ranking the top-50 documents, DeBERTaV3 outperforms SPLADE-v3 on most datasets, but MiniLM shows comparable performance, indicating minimal benefit from re-ranking with this model.
• SPLADE-v3-Lexical achieves high performance on MS MARCO and LoTTE but underperforms on BEIR, highlighting the importance of query encoding for out-of-domain generalization.