[Paper Review] Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold

Swabha Swayamdipta, Sam Thomson+2 more|arXiv (Cornell University)|Jun 29, 2017

Natural Language Processing Techniques46 references95 citations

TL;DR

Introduces a recall-focused softmax-margin SegRNN for frame-semantic argument identification without syntax, then adds syntactic features or a training-time syntactic scaffold to achieve state-of-the-art results without parsing at test time.

ABSTRACT

We present a new, efficient frame-semantic parser that labels semantic arguments to FrameNet predicates. Built using an extension to the segmental RNN that emphasizes recall, our basic system achieves competitive performance without any calls to a syntactic parser. We then introduce a method that uses phrase-syntactic annotations from the Penn Treebank during training only, through a multitask objective; no parsing is required at training or test time. This "syntactic scaffold" offers a cheaper alternative to traditional syntactic pipelining, and achieves state-of-the-art performance.

Motivation & Objective

• Develop a syntax-free frame-semantic argument identification model using segmental RNNs (SegRNNs).
• Evaluate how adding syntactic features (dependency/phrase-structure) affects performance.
• Propose and assess a syntactic scaffold multitask objective trained on Penn Treebank to mimic syntax usefulness without parsing at test time.
• Demonstrate state-of-the-art or near state-of-the-art results on FrameNet 1.5 with and without syntax.
• Release open-source implementation (open-sesame) for reproducibility and broader use.

Proposed method

• Use a SegRNN (Semi-Markov CRF with span representations from bidirectional LSTMs) to score labeled spans for each frame’s arguments.
• Represent spans with span embeddings derived from a token-level biLSTM and a second biLSTM over candidate spans.
• Incorporate frame, lexical unit, and target context embeddings to form segment scores via a feedforward network (Eq. 7).
• Train with a softmax-margin objective to bias recall over precision (Eqs. 8–12).
• Optionally augment with syntax by adding features from (a) a dependency parser or (b) a phrase-structure parser (RNNG) to the span representation (Table 1–2).
• Introduce a syntactic scaffold multitask objective (predict span constituents on PTB) trained alongside the frame-semantic model (Eq. 16); no parsing is required at test time.

Experimental results

Research questions

• RQ1 Can a syntax-free SegRNN identify frame-semantic arguments with competitive accuracy?
• RQ2 Do standard syntactic features (dependency/phrase-structure) provide performance gains when combined with a neural SegRNN?
• RQ3 Can a syntactic scaffold learned from PTB improve frame-semantic parsing without requiring syntactic parsing during training or testing?
• RQ4 How does recall-oriented training (softmax-margin) compare to plain log-likelihood for this task?
• RQ5 What is the relative performance of the proposed approaches compared to state-of-the-art systems on FrameNet 1.5?

Key findings

• The basic syntax-free SegRNN (Open-SESAME) outperforms the SEMAFOR baseline without any syntactic cues.
• Adding syntactic features (phrase-structure or dependency) improves performance, with comparable gains from either type.
• Syntactic scaffold training achieves state-of-the-art performance without any syntactic parser at train or test time, closely matching models that use syntax.
• Softmax-margin recall-oriented training consistently outperforms plain log-loss across all models (Fig. 3).
• Self-ensembling yields notable gains and brings the best ensemble results to within 0.2% F1 of the state-of-the-art when compared to top systems using external resources.