[Paper Review] Automatic Context Pattern Generation for Entity Set Expansion

Yinghui Li, Shulin Huang+7 more|arXiv (Cornell University)|Jul 17, 2022

Topic Modeling4 citations

TL;DR

This paper proposes GAPA, a corpus-independent Entity Set Expansion framework that generates high-quality context patterns using two reverse-direction GPT-2 models, enabling entity expansion without reliance on annotated corpora. The method achieves competitive performance on benchmark datasets by leveraging automatically generated context patterns to guide iterative entity expansion through similarity scoring.

ABSTRACT

Entity Set Expansion (ESE) is a valuable task that aims to find entities of the target semantic class described by given seed entities. Various Natural Language Processing (NLP) and Information Retrieval (IR) downstream applications have benefited from ESE due to its ability to discover knowledge. Although existing corpus-based ESE methods have achieved great progress, they still rely on corpora with high-quality entity information annotated, because most of them need to obtain the context patterns through the position of the entity in a sentence. Therefore, the quality of the given corpora and their entity annotation has become the bottleneck that limits the performance of such methods. To overcome this dilemma and make the ESE models free from the dependence on entity annotation, our work aims to explore a new ESE paradigm, namely corpus-independent ESE. Specifically, we devise a context pattern generation module that utilizes autoregressive language models (e.g., GPT-2) to automatically generate high-quality context patterns for entities. In addition, we propose the GAPA, a novel ESE framework that leverages the aforementioned GenerAted PAtterns to expand target entities. Extensive experiments and detailed analyses on three widely used datasets demonstrate the effectiveness of our method. All the codes of our experiments are available at https://github.com/geekjuruo/GAPA.

Motivation & Objective

• To address the limitation of corpus-based ESE methods that depend heavily on high-quality, manually annotated corpora.
• To eliminate the bottleneck of entity annotation quality by enabling ESE without requiring annotated training corpora.
• To explore a new paradigm—corpus-independent ESE—where context patterns are automatically generated via pre-trained language models.
• To improve entity expansion performance by leveraging context patterns that capture fine-grained semantic distinctions.
• To develop a framework that dynamically enhances supervision signals through iterative seed set refinement.

Proposed method

• The method employs two separate GPT-2 models—one autoregressive from left to right and one from right to left—to generate context patterns (prev-text and next-text) around each entity.
• The reverse-direction GPT-2 model is fine-tuned on a reversed corpus to enable coherent right-to-left text generation, ensuring high-quality context pattern generation.
• Context representations are derived from the generated patterns, and entity similarity is computed based on the assumption that similar entities share similar contexts.
• An iterative expansion process scores candidate entities by their context similarity to seed entities and adds top-ranked entities to the seed set in each round.
• A supervision signal enhancement module updates the seed and candidate sets using entity similarity, mitigating sparse supervision issues.
• The framework is fully end-to-end and does not require any annotated corpora or entity annotations, enabling corpus-independent operation.

Experimental results

Research questions

• RQ1 Can context patterns for entity set expansion be effectively generated without relying on annotated corpora?
• RQ2 Can reverse-direction autoregressive language models generate fluent and relevant context patterns for entities?
• RQ3 Does a corpus-independent ESE framework using automatically generated patterns outperform traditional corpus-based methods?
• RQ4 How well can the generated patterns distinguish fine-grained semantic classes, such as provinces vs. cities within a country?
• RQ5 What are the failure modes of the method, particularly regarding nested or ambiguous entity mentions?

Key findings

• The GAPA framework achieves competitive performance on three widely used ESE benchmarks without requiring any annotated corpora or entity annotations.
• The context pattern generation module produces high-quality, fluent, and relevant patterns, as demonstrated by qualitative case studies on diverse semantic classes.
• The use of bidirectional GPT-2 models enables the generation of context patterns that implicitly encode related entities, enhancing semantic representation.
• The method effectively avoids incorrect expansions into fine-grained semantic subclasses, such as distinguishing 'Anhui' (province) from 'Nanjing' (city) in the China Provinces class.
• The model occasionally expands nested entities like 'Mississippi River' due to context pattern noise, indicating a need for improved decoding strategies to reduce false positives.
• Empirical results confirm that the supervision signal enhancement module effectively mitigates sparse supervision, improving iterative expansion stability and accuracy.