[Paper Review] Positional Attention-based Frame Identification with BERT: A Deep Learning Approach to Target Disambiguation and Semantic Frame Selection

Sang-Sang Tan, Jin‐Cheon Na|arXiv (Cornell University)|Oct 31, 2019

Topic Modeling49 references4 citations

TL;DR

PAFIBERT is a BERT-based deep learning model that uses a positional attention mechanism to improve frame identification in frame-semantic parsing by attending to target-specific contexts, achieving state-of-the-art performance on both in-domain and out-of-domain benchmarks, with superior handling of ambiguous targets and unseen lexical units without frame filtering.

ABSTRACT

Semantic parsing is the task of transforming sentences from natural language into formal representations of predicate-argument structures. Under this research area, frame-semantic parsing has attracted much interest. This parsing approach leverages the lexical information defined in FrameNet to associate marked predicates or targets with semantic frames, thereby assigning semantic roles to sentence components based on pre-specified frame elements in FrameNet. In this paper, a deep neural network architecture known as Positional Attention-based Frame Identification with BERT (PAFIBERT) is presented as a solution to the frame identification subtask in frame-semantic parsing. Although the importance of this subtask is well-established, prior research has yet to find a robust solution that works satisfactorily for both in-domain and out-of-domain data. This study thus set out to improve frame identification in light of recent advancements of language modeling and transfer learning in natural language processing. The proposed method is partially empowered by BERT, a pre-trained language model that excels at capturing contextual information in texts. By combining the language representation power of BERT with a position-based attention mechanism, PAFIBERT is able to attend to target-specific contexts in sentences for disambiguating targets and associating them with the most suitable semantic frames. Under various experimental settings, PAFIBERT outperformed existing solutions by a significant margin, achieving new state-of-the-art results for both in-domain and out-of-domain benchmark test sets.

Motivation & Objective

• Address the challenge of robust frame identification in frame-semantic parsing, especially for ambiguous targets and out-of-domain data.
• Leverage pre-trained language models like BERT to enhance contextual representation for target disambiguation.
• Develop a method that effectively distinguishes multiple instances of the same target in a sentence by attending to their distinct positional contexts.
• Improve generalization to unseen targets and non-contiguous multi-word lexical units by minimizing reliance on strict frame filtering.
• Balance the trade-off between candidate frame constraints and contextual flexibility to enhance model robustness and accuracy.

Proposed method

• Utilizes BERT as the contextual encoder to generate deep contextual representations of input sentences.
• Introduces a position-based attention mechanism that allows the model to attend selectively to context around each target instance based on its position in the sentence.
• Applies a multi-head attention mechanism over the BERT-embedded sequence to focus on relevant context for each target, improving disambiguation of homographic targets.
• Employs two variants: one with frame filtering (using FrameNet’s candidate frames) and one without, to evaluate trade-offs in robustness and generalization.
• Fine-tunes the BERT encoder with a classification head to predict the most appropriate semantic frame for each target.
• Uses cross-entropy loss during training to optimize frame prediction for each target instance.

Experimental results

Research questions

• RQ1 Can a BERT-based model with positional attention improve frame identification accuracy over existing methods on both in-domain and out-of-domain test sets?
• RQ2 How does frame filtering affect the model's ability to generalize to unseen targets and non-contiguous multi-word lexical units?
• RQ3 To what extent can the model resolve subtle semantic distinctions between highly confused frames without relying on frame constraints?
• RQ4 Does the position-based attention mechanism effectively disambiguate multiple occurrences of the same target in a single sentence?
• RQ5 Can a frame-agnostic variant of the model outperform frame-filtered counterparts in handling domain-specific or rare lexical uses?

Key findings

• PAFIBERT achieved new state-of-the-art results on both in-domain and out-of-domain benchmark datasets for frame identification.
• The variant without frame filtering outperformed the filtered version in handling non-contiguous multi-word lexical units, such as 'give up.v' with 'it' inserted between 'give' and 'up'.
• PAFIBERT without frame filtering correctly identified 'highlighting' as [Body_decoration] in a beauty domain context, demonstrating domain-aware frame selection.
• The model showed improved performance on ambiguous targets by attending to context-specific features through the position-based attention mechanism.
• Despite its advantages, the no-filtering variant occasionally misclassified highly ambiguous targets (e.g., 'worth' predicted as [Earnings_and_losses] instead of [Deserving]), indicating sensitivity to contextual variation.
• The most frequent frame confusion pairs were found to be a major source of accuracy loss, suggesting that future work should focus on resolving these nuanced distinctions.