[Paper Review] Improved Semantic Role Labeling using Parameterized Neighborhood Memory Adaptation

Ishan Jindal, Ranit Aharonov+3 more|arXiv (Cornell University)|Nov 29, 2020

Natural Language Processing Techniques27 references4 citations

TL;DR

This paper proposes Parameterized Neighborhood Memory Adaptation (PNMA), a method that enhances semantic role labeling (SRL) by leveraging parameterized representations of the K nearest neighbors from a memory of deep neural network activations. By retraining classification layers using these adaptive neighborhood vectors, PNMA improves SRL performance across diverse settings, achieving state-of-the-art F1 scores of 80.2% on CoNLL2005 and 84.97% on CoNLL2009, especially benefiting low-frequency and out-of-domain predicates when combined with BERT.

ABSTRACT

Deep neural models achieve some of the best results for semantic role labeling. Inspired by instance-based learning that utilizes nearest neighbors to handle low-frequency context-specific training samples, we investigate the use of memory adaptation techniques in deep neural models. We propose a parameterized neighborhood memory adaptive (PNMA) method that uses a parameterized representation of the nearest neighbors of tokens in a memory of activations and makes predictions based on the most similar samples in the training data. We empirically show that PNMA consistently improves the SRL performance of the base model irrespective of types of word embeddings. Coupled with contextualized word embeddings derived from BERT, PNMA improves over existing models for both span and dependency semantic parsing datasets, especially on out-of-domain text, reaching F1 scores of 80.2, and 84.97 on CoNLL2005, and CoNLL2009 datasets, respectively.

Motivation & Objective

• To improve semantic role labeling (SRL) performance by leveraging nearest neighbor information from deep neural network activations.
• To address the challenge of low-frequency predicates and out-of-domain generalization in SRL.
• To develop a memory-based adaptation method that enhances base models without requiring labeled neighbors.
• To demonstrate the effectiveness of PNMA across various word embedding types, including contextualized embeddings like BERT.

Proposed method

• Construct a memory of token representations from the training set, storing the K nearest neighbors for each token using Euclidean distance.
• Generate a parameterized neighborhood representation $\mathbf{n}_{K}(w) \in \mathbb{R}^d$ by combining the K nearest neighbor representations using learnable weights.
• Fix the base model's word and predicate embeddings and LSTM layers, and fine-tune only the classification layer using the neighborhood representation $\mathbf{n}_{K}(w)$.
• Use the neighborhood representation to retrain the classification layer, enabling the model to correct mispredictions by the base model.
• Apply PNMA to both span and dependency-style SRL tasks, integrating it with models using static and contextualized word embeddings.

Experimental results

Research questions

• RQ1 Can nearest neighbor information from deep neural network activations improve SRL performance beyond the base model?
• RQ2 Does PNMA effectively reduce errors on low-frequency predicates and out-of-domain examples?
• RQ3 How does PNMA perform when combined with contextualized embeddings like BERT?
• RQ4 Can PNMA improve predictions on samples where the base model and PNMA disagree?
• RQ5 What is the impact of neighborhood size and representation quality on PNMA's performance?

Key findings

• PNMA achieves a new state-of-the-art F1 score of 80.2% on the CoNLL2005 out-of-domain test set, representing a 2.0-point improvement over prior methods.
• On the CoNLL2009 dataset, PNMA with BERT achieves an F1 score of 84.97%, marking a 5.0-point improvement over existing models.
• PNMA consistently improves the base model’s performance across all argument roles, especially for low-frequency predicates and rare syntactic patterns.
• The method corrects the base model’s predictions in cases of disagreement, with PNMA being correct more often than the base model on both CoNLL2005 and CoNLL2009.
• Error analysis shows PNMA most benefits samples with moderate neighborhood size (10–40 neighbors) and low predicate frequency, but struggles when neighbors have incorrect base model predictions.