[Paper Review] Features and Aggregators for Web-scale Entity Search

Uma Sawant, Soumen Chakrabarti|arXiv (Cornell University)|Mar 13, 2013

Topic Modeling29 references3 citations

TL;DR

This paper proposes a unified, trainable framework for web-scale entity search that jointly optimizes snippet-level scoring and entity-level aggregation using discriminative learning. It introduces a universal feature encoding for query-term perplexity and proximity to entity mentions, and directly trains on entity-level relevance, outperforming prior systems by 8% in MAP on a large, noisy Web dataset while matching or exceeding state-of-the-art performance on TREC.

ABSTRACT

We focus on two research issues in entity search: scoring a document or snippet that potentially supports a candidate entity, and aggregating scores from different snippets into an entity score. Proximity scoring has been studied in IR outside the scope of entity search. However, aggregation has been hardwired except in a few cases where probabilistic language models are used. We instead explore simple, robust, discriminative ranking algorithms, with informative snippet features and broad families of aggregation functions. Our first contribution is a study of proximity-cognizant snippet features. In contrast with prior work which uses hardwired "proximity kernels" that implement a fixed decay with distance, we present a "universal" feature encoding which jointly expresses the perplexity (informativeness) of a query term match and the proximity of the match to the entity mention. Our second contribution is a study of aggregation functions. Rather than train the ranking algorithm on snippets and then aggregate scores, we directly train on entities such that the ranking algorithm takes into account the aggregation function being used. Our third contribution is an extensive Web-scale evaluation of the above algorithms on two data sets having quite different properties and behavior. The first one is the W3C dataset used in TREC-scale enterprise search, with pre-annotated entity mentions. The second is a Web-scale open-domain entity search dataset consisting of 500 million Web pages, which contain about 8 billion token spans annotated automatically with two million entities from 200,000 entity types in Wikipedia. On the TREC dataset, the performance of our system is comparable to the currently prevalent systems. On the much larger and noisier Web dataset, our system delivers significantly better performance than all other systems, with 8% MAP improvement over the closest competitor.

Motivation & Objective

• To address the lack of unified, trainable approaches for scoring snippets and aggregating evidence in entity search.
• To develop a robust, discriminative framework that directly optimizes for entity-level relevance instead of snippet-level relevance.
• To evaluate the framework on diverse datasets, including a large-scale, noisy Web dataset and a TREC enterprise search dataset.
• To investigate the impact of different aggregation functions and feature designs on entity ranking performance.
• To demonstrate scalability and effectiveness of the system on 500 million Web pages with two million entities.

Proposed method

• Proposes a universal feature encoding that jointly models query term perplexity (via inverse document frequency) and lexical proximity to the entity mention, replacing fixed proximity kernels.
• Introduces a pairwise ranking loss function that trains the model directly on entity-level relevance judgments, incorporating various aggregation functions as part of the learning process.
• Employs a family of aggregation functions—such as sum, mean, soft cutoff, and soft max—treated as trainable components within the learning framework.
• Uses discriminative learning-to-rank (L2R) with max-margin or conditional probability formulations, avoiding reliance on generative language models.
• Applies symmetric aggregation (e.g., sum) over context scores, avoiding asymmetric or rank-based cutoffs that degrade performance on noisy data.
• Performs offline entity annotation and context scoring, enabling efficient query-time inference over large-scale Web data.

Experimental results

Research questions

• RQ1 Can a unified, trainable feature representation that combines perplexity and proximity outperform fixed proximity kernels in entity search?
• RQ2 Does direct end-to-end training on entity-level relevance judgments improve performance compared to snippet-level training with hardwired aggregation?
• RQ3 How do different aggregation functions (e.g., sum, mean, soft cutoff) impact entity ranking performance on diverse datasets?
• RQ4 Is symmetric aggregation (e.g., sum) more robust than asymmetric or rank-based cutoffs in noisy, large-scale Web environments?
• RQ5 To what extent do features like context count |S_e| contribute to performance on different data distributions?

Key findings

• On the large, noisy Web dataset of 500 million pages and 2 million entities, the proposed system achieved an 8% improvement in Mean Average Precision (MAP) over the closest competitor.
• On the TREC dataset, the system performed comparably to state-of-the-art systems by Balog et al. and MacDonald et al., with competitive results across all metrics including MRR and NDCG.
• Simple rectangle-based features encoding both perplexity and proximity outperformed proximity kernels combined with probabilistic language models.
• The sum aggregation function consistently outperformed averaging, soft cutoff, SoftOr, and SoftMax across both datasets, indicating that evidence should be summed rather than normalized.
• Rank-based or asymmetric cutoffs degraded performance on the Web dataset, suggesting that symmetric aggregation is more robust in noisy environments.
• The feature |S_e| (number of supporting contexts) was valuable on TREC but not on the Web dataset, indicating dataset-dependent feature utility.