[Paper Review] UHD-BERT: Bucketed Ultra-High Dimensional Sparse Representations for Full Ranking.

Kyoung-Rok Jang, Junmo Kang+5 more|arXiv (Cornell University)|Apr 15, 2021

Topic Modeling24 references4 citations

TL;DR

UHD-BERT proposes a neuro-symbolic information retrieval framework that uses ultra-high dimensional (UHD) sparse representations derived from BERT to balance semantic richness with efficiency. By applying bucketing to combine horizontal (embedding segments) or vertical (BERT layer outputs) components, it achieves disentangled, interpretable, and efficient representations compatible with symbolic IR techniques like inverted indexing and BM25.

ABSTRACT

Neural information retrieval (IR) models are promising mainly because their semantic matching capabilities can ameliorate the well-known synonymy and polysemy problems of word-based symbolic approaches. However, the power of neural models' dense representations comes at the cost of inefficiency, limiting it to be used as a re-ranker. Sparse representations, on the other hand, can help enhance symbolic or latent-term representations and yet take advantage of an inverted index for efficiency, being amenable to symbolic IR techniques that have been around for decades. In order to transcend the trade-off between sparse representations (symbolic or latent-term based) and dense representations, we propose an ultra-high dimensional (UHD) representation scheme equipped with directly controllable sparsity. With the high dimensionality, we attempt to make the meaning of each dimension less entangled and polysemous than dense embeddings. The sparsity allows for not only efficiency for vector calculations but also the possibility of making individual dimensions attributable to interpretable concepts. Our model, UHD-BERT, maximizes the benefits of ultra-high dimensional (UHD) sparse representations based on BERT language modeling, by adopting a bucketing method. With this method, different segments of an embedding (horizontal buckets) or the embeddings from multiple layers of BERT (vertical buckets) can be selected and merged so that diverse linguistic aspects can be represented. An additional and important benefit of our highly disentangled (high-dimensional) and efficient (sparse) representations is that this neural approach can be harmonized with well-studied symbolic IR techniques (e.g., inverted index, pseudo-relevance feedback, BM25), enabling us to build a powerful and efficient neuro-symbolic information retrieval system.

Motivation & Objective

• Address the trade-off between dense neural embeddings (semantic but inefficient) and sparse symbolic representations (efficient but limited in semantic expressiveness).
• Enable sparse representations that retain semantic richness by increasing dimensionality while maintaining sparsity for computational efficiency.
• Make individual vector dimensions interpretable by reducing polysemy and entanglement through ultra-high dimensionality.
• Integrate neural representations with established symbolic IR techniques such as inverted indexing, BM25, and pseudo-relevance feedback.
• Develop a scalable and efficient full-ranking system that combines the strengths of deep learning and classical IR methods.

Proposed method

• Propose an ultra-high dimensional (UHD) sparse representation scheme derived from BERT, where each dimension corresponds to a disentangled, low-entanglement semantic unit.
• Apply a bucketing mechanism to select and merge specific segments of BERT embeddings (horizontal buckets) or outputs from different BERT layers (vertical buckets) to capture diverse linguistic aspects.
• Control sparsity directly by selecting only the top-k most salient dimensions per bucket, enabling efficient vector operations and compatibility with inverted indices.
• Construct a neuro-symbolic retrieval system where UHD-BERT embeddings are indexed using an inverted index, allowing fast retrieval and integration with classical IR models.
• Harmonize UHD-BERT with symbolic techniques such as BM25 and pseudo-relevance feedback by combining sparse neural features with traditional term-level features in a unified ranking framework.

Experimental results

Research questions

• RQ1 Can ultra-high dimensional sparse representations from BERT achieve competitive ranking performance while maintaining computational efficiency?
• RQ2 To what extent can bucketing of BERT embeddings (horizontal or vertical) improve the disentanglement and interpretability of semantic features?
• RQ3 How well can UHD-BERT integrate with symbolic IR techniques like inverted indexing and BM25 in a full-ranking setting?
• RQ4 Does the combination of sparse, high-dimensional neural features with classical IR methods outperform dense neural re-rankers in terms of both accuracy and efficiency?

Key findings

• UHD-BERT achieves competitive or superior ranking performance compared to dense neural re-rankers while maintaining efficiency through sparsity and inverted indexing.
• The bucketing mechanism enables effective representation of diverse linguistic aspects by combining specific embedding segments or multi-layer BERT outputs.
• High dimensionality reduces polysemy and entanglement, making individual vector dimensions more interpretable and semantically distinct.
• The integration of UHD-BERT with symbolic IR techniques such as BM25 and pseudo-relevance feedback enhances retrieval effectiveness in a neuro-symbolic framework.
• The system supports full-ranking at scale due to the compatibility of sparse representations with standard IR infrastructure, including inverted indices and efficient vector search.