[Paper Review] Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention

Ekta Sood, Simon Tannert+2 more|arXiv (Cornell University)|Oct 15, 2020

Topic ModelingComputer Science85 references32 citations

TL;DR

This paper introduces a hybrid Text Saliency Model (TSM) that blends a cognitive reading model with real gaze supervision and a joint training approach to integrate gaze-informed attention into NLP tasks, achieving strong gains in paraphrase generation and competitive results in sentence compression.

ABSTRACT

A lack of corpora has so far limited advances in integrating human gaze data as a supervisory signal in neural attention mechanisms for natural language processing(NLP). We propose a novel hybrid text saliency model(TSM) that, for the first time, combines a cognitive model of reading with explicit human gaze supervision in a single machine learning framework. On four different corpora we demonstrate that our hybrid TSM duration predictions are highly correlated with human gaze ground truth. We further propose a novel joint modeling approach to integrate TSM predictions into the attention layer of a network designed for a specific upstream NLP task without the need for any task-specific human gaze data. We demonstrate that our joint model outperforms the state of the art in paraphrase generation on the Quora Question Pairs corpus by more than 10% in BLEU-4 and achieves state of the art performance for sentence compression on the challenging Google Sentence Compression corpus. As such, our work introduces a practical approach for bridging between data-driven and cognitive models and demonstrates a new way to integrate human gaze-guided neural attention into NLP tasks.

Motivation & Objective

• Motivate the use of human gaze as supervision for neural attention in NLP due to data scarcity.
• Propose a hybrid Text Saliency Model (TSM) that leverages a cognitive reading model to produce gaze-like saliency with synthetic data.
• Develop a joint modeling framework to integrate TSM predictions into attention mechanisms for downstream NLP tasks without requiring task-specific gaze data.
• Demonstrate state-of-the-art or competitive improvements on paraphrase generation and sentence compression using the joint model.

Proposed method

• Hybrid text saliency model (TSM) combines a BiLSTM embedding layer with a Transformer encoder to predict word-level fixation durations.
• TSM is pre-trained on synthetic data generated by the E-Z Reader cognitive model and fine-tuned on real gaze data from Provo and GECO corpora.
• A joint modeling approach modifies the Luong attention mechanism to multiply attention scores by the TSM predictions, allowing end-to-end training with task-specific losses.
• The upstream tasks studied are paraphrase generation (Quora dataset) and sentence compression (Google Sentence Compression dataset).
• TSM is trained with mean squared error to predict fixation durations and then adapted during joint training to the specific downstream task.

Experimental results

Research questions

• RQ1 Can a hybrid cognitive/data-driven text saliency model produce gaze-like predictions suitable for guiding neural attention in NLP?
• RQ2 Does integrating TSM predictions into task-specific attention layers improve downstream NLP performance without requiring task-specific gaze data?
• RQ3 How does pre-training on cognitive gaze simulations plus real gaze fine-tuning affect generalization to out-of-domain data?
• RQ4 What is the impact of adapting TSM during joint training on paraphrase generation and sentence compression performance?

Key findings

• TSM duration predictions correlate with human gaze ground truth on out-of-domain data.
• Joint model with TSM-attention outperforms state-of-the-art in paraphrase generation on Quora by over 10% BLEU-4 (47? not stated; ground truth shows BLEU-4 improvements to 28.82 from 17.9 in prior work with 100K data) and achieves competitive sentence compression results (F1 ~85.0, CR ~0.39).
• Ablation studies show full model consistently outperforms No Fixation, Random TSM Init, TSM Weight Swap, and Frozen TSM variants, evidencing task-specific adaptation of saliency.
• Pre-training on E-Z Reader synthetic data plus fine-tuning on real gaze data significantly improves saliency prediction metrics (MSE, JSD, Spearman ρ) across multiple corpora, especially in smaller datasets like Provo.
• The adaptation of TSM during joint training yields attention distributions that differ by task and align better with human-like focus, enabling task-specific saliency.