[Paper Review] Open Domain Web Keyphrase Extraction Beyond Language Modeling

Lee Xiong, Chuan Hu+3 more|arXiv (Cornell University)|Nov 6, 2019

Advanced Text Analysis Techniques27 references4 citations

TL;DR

This paper introduces OpenKP, a large-scale open-domain keyphrase extraction dataset with 100,000 web documents and expert-annotated keyphrases, and proposes BLING-KPE, a neural model that enhances keyphrase extraction beyond language modeling by integrating visual document features and weak supervision from search query clicks. BLING-KPE significantly outperforms existing methods and demonstrates strong zero-shot generalization to unseen domains like news, outperforming even specialized neural models trained on scientific data.

ABSTRACT

This paper studies keyphrase extraction in real-world scenarios where documents are from diverse domains and have variant content quality. We curate and release OpenKP, a large scale open domain keyphrase extraction dataset with near one hundred thousand web documents and expert keyphrase annotations. To handle the variations of domain and content quality, we develop BLING-KPE, a neural keyphrase extraction model that goes beyond language understanding using visual presentations of documents and weak supervision from search queries. Experimental results on OpenKP confirm the effectiveness of BLING-KPE and the contributions of its neural architecture, visual features, and search log weak supervision. Zero-shot evaluations on DUC-2001 demonstrate the improved generalization ability of learning from the open domain data compared to a specific domain.

Motivation & Objective

• To address the lack of large-scale, diverse, real-world web document datasets for keyphrase extraction in non-scientific domains.
• To improve keyphrase extraction performance on low-resource, high-variability web documents with diverse content quality and structure.
• To develop a model that generalizes well across domains by leveraging visual layout and search behavior signals beyond pure language understanding.
• To demonstrate that weak supervision from search click logs and visual features can significantly enhance keyphrase extraction in open-domain settings.

Proposed method

• Curated OpenKP, a dataset of ~100,000 web documents from diverse domains with expert-annotated keyphrases, publicly released at aka.ms/BLING.
• Proposed BLING-KPE, a neural model using a convolutional transformer architecture to jointly model linguistic and visual features of web documents.
• Integrated visual features—such as font size, position, and HTML structure—into word embeddings to capture document layout salience.
• Introduced a pre-training task called 'Query Prediction' using search click logs to weakly supervise keyphrase relevance at scale.
• Combined contextual embeddings (e.g., ELMo), visual features, and click-based supervision to improve keyphrase detection beyond language modeling.
• Applied heuristics (weighted sum and deduplication) to merge keyphrases from document chunks during zero-shot evaluation on DUC-2001.

Experimental results

Research questions

• RQ1 Can a keyphrase extraction model trained on diverse, real-world web documents generalize effectively to unseen domains like news, without fine-tuning?
• RQ2 To what extent do visual document features and search click logs contribute to keyphrase extraction performance beyond language modeling?
• RQ3 How does BLING-KPE compare to state-of-the-art neural and non-neural keyphrase extraction methods on open-domain and news datasets?
• RQ4 Does pre-training on search click logs as weak supervision improve keyphrase extraction accuracy and robustness across domains?
• RQ5 What is the relative contribution of language modeling, visual features, and click-based supervision in the overall performance of BLING-KPE?

Key findings

• BLING-KPE outperforms standard KPE baselines, recent neural models, and a highly optimized commercial system on the OpenKP dataset with large margins.
• Ablation studies show that removing visual features or search click pretraining significantly reduces model accuracy, confirming their critical role.
• BLING-KPE trained on OpenKP achieves the highest F1@10 score on DUC-2001 in zero-shot evaluation, outperforming TFIDF and all other neural models.
• The model trained on OpenKP generalizes better than models trained on scientific-domain data, which fail to generalize to news documents.
• Visual features help identify salient phrases like product names and types that are visually emphasized, even when they are not top-ranked by language models alone.
• The Query Prediction pre-training task effectively leverages large-scale search logs to provide meaningful supervision signals, improving keyphrase relevance detection.