[Paper Review] Advancing NLP with Cognitive Language Processing Signals

Nora Hollenstein, Maria Barrett+4 more|arXiv (Cornell University)|Apr 4, 2019

Topic Modeling49 references34 citations

TL;DR

The paper investigates whether gaze (eye-tracking) and EEG signals can consistently improve NLP tasks across multiple tasks and data sources, and whether combining signals yields additional benefits.

ABSTRACT

When we read, our brain processes language and generates cognitive processing data such as gaze patterns and brain activity. These signals can be recorded while reading. Cognitive language processing data such as eye-tracking features have shown improvements on single NLP tasks. We analyze whether using such human features can show consistent improvement across tasks and data sources. We present an extensive investigation of the benefits and limitations of using cognitive processing data for NLP. Specifically, we use gaze and EEG features to augment models of named entity recognition, relation classification, and sentiment analysis. These methods significantly outperform the baselines and show the potential and current limitations of employing human language processing data for NLP.

Motivation & Objective

• Motivate the use of cognitive processing signals (gaze and EEG) to improve NLP beyond single-task, single-signal setups.
• Evaluate improvements across multiple NLP tasks (NER, relation classification, sentiment analysis) and data sources.
• Investigate methods to incorporate signals into models, including test-time data-agnostic approaches and multi-task learning.
• Assess practicality, generalizability, and potential limitations of cognitive signals for NLP.

Proposed method

• Use ZuCo corpus with simultaneous eye-tracking and EEG data for English sentence reading.
• Augment state-of-the-art models for NER, relation classification, and sentiment analysis with gaze and EEG features.
• Explore three integration approaches: add features to baselines, generalize features for test-time-free prediction via type-aggregated features, and employ multi-task learning with cognitive features as auxiliary tasks.
• Normalize and align gaze/EEG features with word/token embeddings and concatenate them to embeddings for downstream models.
• Perform extensive cross-validation and significance testing (permutation tests with Bonferroni correction) to assess improvements.

Experimental results

Research questions

• RQ1 Do gaze and EEG signals provide consistent improvements across NER, relation classification, and sentiment analysis tasks?
• RQ2 Is combining gaze and EEG signals more beneficial than using either signal alone?
• RQ3 Can cognitive signals be generalized so that test-time data collection is not required (type-aggregated features)?
• RQ4 What is the impact of cognitive signals when used as auxiliary tasks in multi-task learning for NLP?

Key findings

• Cognitive features yield consistent improvements over baselines across all three NLP tasks.
• Gaze features and EEG features each improve performance, but combining them does not surpass the best single-signal results.
• Type-aggregated (word type-level) features improve performance without requiring real-time cognitive data at test time, though EEG-type aggregation shows mixed effects on external corpora.
• Permutation-based significance tests indicate improvements are statistically significant in the majority of configurations after Bonferroni correction (11/12 configurations).
• Single-subject models can outperform averaged models in some cases, but averaging often yields competitive results and robustness.