[Paper Review] KPEval: Towards Fine-Grained Semantic-Based Keyphrase Evaluation

Di Wu, Da Yin+1 more|arXiv (Cornell University)|Mar 27, 2023

Advanced Text Analysis Techniques4 citations

TL;DR

KPEval proposes a fine-grained, semantic-based evaluation framework for keyphrase systems that moves beyond exact matching by assessing four dimensions: reference agreement, faithfulness, diversity, and utility. Using semantic embeddings and retrieval-based metrics, it demonstrates superior alignment with human preferences and reveals that large language models like GPT-3.5 outperform prior state-of-the-art models, while no single model excels across all evaluation aspects.

ABSTRACT

Despite the significant advancements in keyphrase extraction and keyphrase generation methods, the predominant approach for evaluation mainly relies on exact matching with human references. This scheme fails to recognize systems that generate keyphrases semantically equivalent to the references or diverse keyphrases that carry practical utility. To better assess the capability of keyphrase systems, we propose KPEval, a comprehensive evaluation framework consisting of four critical aspects: reference agreement, faithfulness, diversity, and utility. For each aspect, we design semantic-based metrics to reflect the evaluation objectives. Meta-evaluation studies demonstrate that our evaluation strategy correlates better with human preferences compared to a range of previously proposed metrics. Using KPEval, we re-evaluate 23 keyphrase systems and discover that (1) established model comparison results have blind-spots especially when considering reference-free evaluation; (2) large language models are underestimated by prior evaluation works; and (3) there is no single best model that can excel in all the aspects.

Motivation & Objective

• Address the limitations of exact-match evaluation in keyphrase systems, which fails to recognize semantic equivalence or practical utility.
• Overcome the over-reliance on reference-based evaluation that overlooks critical properties like faithfulness, diversity, and real-world utility.
• Develop a holistic evaluation framework that aligns model performance with application-specific needs.
• Demonstrate that existing evaluation metrics correlate poorly with human preferences and that semantic-based metrics significantly improve this correlation.
• Re-evaluate 21 keyphrase systems to uncover blind spots in prior model comparisons and reveal underappreciated strengths of large language models.

Proposed method

• Design a phrase-level semantic matching metric using a high-quality, large-scale keyphrase-specific embedding model trained on KP20k and KPTimes.
• Introduce a reference agreement metric based on semantic F1 (SemF1) that computes similarity between predicted and reference keyphrases using dense semantic embeddings.
• Propose a faithfulness metric that evaluates whether predicted keyphrases are grounded in the input document using a sequence-to-sequence consistency model.
• Develop a diversity metric based on embedding similarity (emb_sim) to measure the conceptual distinctness of predicted keyphrases.
• Implement a utility evaluation using dense retrieval to simulate downstream information retrieval performance, measuring Mean Reciprocal Rank (RR) of retrieved documents.
• Conduct meta-evaluation on five keyphrase systems using human-annotated preference data to validate the correlation of KPEval metrics with human judgments.

Figure 1: An illustration of the proposed framework. KPEval evaluates keyphrase systems on four crucial properties and incorporates semantic-based metrics for more accurate assessment.

Experimental results

Research questions

• RQ1 How well do existing reference-based evaluation metrics correlate with human preferences compared to the proposed semantic-based metrics?
• RQ2 To what extent do current evaluation practices fail to detect models that generate semantically correct or diverse keyphrases?
• RQ3 How do large language models such as GPT-3.5 perform across multiple evaluation dimensions when assessed with KPEval?
• RQ4 Is there a single model that performs best across all four evaluation aspects (reference agreement, faithfulness, diversity, utility)?
• RQ5 Can a holistic evaluation framework like KPEval uncover blind spots in established model comparisons and reveal previously overlooked model capabilities?

Key findings

• The proposed semantic-based reference agreement metric (SemF1) achieves a Kendall’s Tau of 0.657 with human preferences, outperforming existing metrics by over 0.15 absolute points.
• Many heuristic improvements to exact matching, such as n-gram or edit-distance relaxation, surprisingly fail to improve human correlation, highlighting the limitations of these approaches.
• ExHiRD-h, previously overlooked in standard benchmarks, demonstrates strong performance across multiple aspects, revealing blind spots in prior model comparisons.
• Large language models like GPT-3.5 significantly outperform existing state-of-the-art keyphrase generation and extraction models, especially in faithfulness and utility, challenging prior conclusions.
• No single model excels across all four evaluation dimensions—different models show strengths in different aspects, indicating the need for application-specific evaluation.
• The utility evaluation via dense retrieval shows that models with higher semantic alignment to the document (e.g., ExHiRD-h) achieve a Mean Reciprocal Rank (RR) of 1.0, indicating perfect retrieval performance for relevant queries.

Figure 2: The 95% confidence intervals for the Kendall’s Tau between human and automatic metrics on KP20k and KPTimes. $SemF1$ exhibits a higher correlation with humans and smaller intervals.