[Paper Review] Parsing Speech: A Neural Approach to Integrating Lexical and Acoustic-Prosodic Information

Trang Tran, Shubham Toshniwal+4 more|arXiv (Cornell University)|Apr 24, 2017

Natural Language Processing Techniques27 references4 citations

TL;DR

This paper proposes a neural encoder-decoder parser that integrates word-level text embeddings with raw acoustic-prosodic features—such as pitch and energy—via a convolutional neural network and attention-based LSTM. It achieves statistically significant improvements in constituent parsing and disfluency detection F1 scores over a strong text-only baseline, especially in disfluent or long sentences, with prosody most effective at correcting attachment errors.

ABSTRACT

In conversational speech, the acoustic signal provides cues that help listeners disambiguate difficult parses. For automatically parsing spoken utterances, we introduce a model that integrates transcribed text and acoustic-prosodic features using a convolutional neural network over energy and pitch trajectories coupled with an attention-based recurrent neural network that accepts text and prosodic features. We find that different types of acoustic-prosodic features are individually helpful, and together give statistically significant improvements in parse and disfluency detection F1 scores over a strong text-only baseline. For this study with known sentence boundaries, error analyses show that the main benefit of acoustic-prosodic features is in sentences with disfluencies, attachment decisions are most improved, and transcription errors obscure gains from prosody.

Motivation & Objective

• To improve parsing of conversational speech by integrating acoustic-prosodic cues without requiring hand-annotated prosodic boundaries.
• To investigate whether raw acoustic features (pitch, energy, pauses) can enhance parsing performance beyond text-only models.
• To analyze where prosodic features provide the most benefit, particularly in disfluent or ambiguous syntactic structures.
• To evaluate the impact of transcription errors on the utility of prosodic features in parsing systems.
• To explore the feasibility of extending this framework to incremental or lattice-based parsing in ASR pipelines.

Proposed method

• The model uses a bidirectional LSTM encoder to process reversed word sequences with concatenated word embeddings and acoustic-prosodic features.
• Acoustic-prosodic features include pitch, energy, and pause bins derived from 1-second windows around each word, normalized per speaker.
• A 1D convolutional layer processes the temporal trajectories of pitch and energy to extract prosodic patterns.
• An attention mechanism attends over the encoder’s hidden states to generate linearized parse outputs step-by-step.
• The decoder uses a deep LSTM with context vectors computed via soft attention over encoder states, conditioned on prior output tokens.
• The framework does not require explicit prosodic boundary annotations, learning end-to-end from raw acoustic and text inputs.

Experimental results

Research questions

• RQ1 Can raw acoustic-prosodic features improve parsing performance in conversational speech when integrated with text embeddings?
• RQ2 In which syntactic contexts (e.g., disfluencies, attachment ambiguities) do prosodic features provide the most benefit?
• RQ3 How do transcription errors affect the observed gains from prosodic features in parsing systems?
• RQ4 Does the integration of prosody yield greater improvements when parsing ASR lattices or alternative hypotheses rather than gold transcripts?
• RQ5 Can this prosody-integration framework be extended to transition-based or dependency parsing architectures?

Key findings

• The inclusion of acoustic-prosodic features leads to statistically significant improvements in F1 scores for both constituent parsing and disfluency detection over a strong text-only baseline.
• The largest gains are observed in disfluent sentences and in correcting attachment errors, indicating prosody helps resolve syntactic ambiguities.
• Individual types of prosodic features (e.g., pitch, energy, pauses) each contribute positively, and their combination yields the best performance.
• When gold transcriptions contain errors, prosodic features can degrade performance due to inconsistency between the acoustic signal and the ground-truth parse, suggesting that prosody may be more beneficial in ASR lattice parsing.
• Error analysis shows that prosody is most effective in repairing syntactic attachment decisions, particularly in complex or interrupted utterances.
• The model’s performance is robust to transcription noise when prosody is included, but gains are obscured when transcripts are inaccurate, highlighting the need for aligned data in future work.