[Paper Review] Identification of Cognitive Workload during Surgical Tasks with Multimodal Deep Learning

Kaizhe Jin, Adrian Rubio-Solis+9 more|arXiv (Cornell University)|Sep 12, 2022

Cardiac, Anesthesia and Surgical Outcomes4 citations

TL;DR

This study proposes a two-stage multimodal deep learning framework that fuses EEG, fNIRS, and pupil diameter signals to identify cognitive workload (CWL) during laparoscopic surgery. Using transfer learning with fine-tuned AlexNet for CWL detection and a 1D-CNN for task-type classification, the model achieves 93% precision in identifying surgical task conditions associated with elevated mental workload, demonstrating high accuracy and low computational cost in a simulated OR setting.

ABSTRACT

The operating room (OR) is a dynamic and complex environment consisting of a multidisciplinary team working together in a high take environment to provide safe and efficient patient care. Additionally, surgeons are frequently exposed to multiple psycho-organisational stressors that may cause negative repercussions on their immediate technical performance and long-term health. Many factors can therefore contribute to increasing the Cognitive Workload (CWL) such as temporal pressures, unfamiliar anatomy or distractions in the OR. In this paper, a cascade of two machine learning approaches is suggested for the multimodal recognition of CWL in four different surgical task conditions. Firstly, a model based on the concept of transfer learning is used to identify if a surgeon is experiencing any CWL. Secondly, a Convolutional Neural Network (CNN) uses this information to identify different degrees of CWL associated to each surgical task. The suggested multimodal approach considers adjacent signals from electroencephalogram (EEG), functional near-infrared spectroscopy (fNIRS) and eye pupil diameter. The concatenation of signals allows complex correlations in terms of time (temporal) and channel location (spatial). Data collection was performed by a Multi-sensing AI Environment for Surgical Task & Role Optimisation platform (MAESTRO) developed at the Hamlyn Centre, Imperial College London. To compare the performance of the proposed methodology, a number of state-of-art machine learning techniques have been implemented. The tests show that the proposed model has a precision of 93%.

Motivation & Objective

• To address the challenge of identifying cognitive workload (CWL) in real-time during high-stakes surgical tasks, where mental overload can impair performance.
• To develop a non-invasive, multimodal physiological monitoring system that captures dynamic changes in brain and autonomic activity during surgery.
• To improve surgical team safety and performance by detecting early signs of cognitive overload through automated, data-driven methods.
• To validate a deep learning pipeline that integrates temporal and spatial correlations across multiple neurophysiological signals (EEG, fNIRS, pupil size) for accurate CWL classification.
• To establish a scalable, low-computational-cost framework for real-time cognitive workload monitoring in clinical environments.

Proposed method

• A two-stage machine learning pipeline is employed: first, transfer learning with a fine-tuned AlexNet model processes scalp signal data transformed into scalogram images to detect the presence of cognitive workload.
• The second stage uses a 1D-CNN to classify the specific surgical task type (four conditions) based on concatenated 1D physiological signals from EEG, fNIRS, and pupil diameter.
• Physiological data is collected via the MAESTRO platform, a multi-sensing AI system developed at Imperial College London’s Hamlyn Centre, capturing signals during simulated laparoscopic peg transfer tasks.
• Signal preprocessing includes transformation into scalograms for visual input in the first stage and 1D vector encoding for the second stage, enabling spatiotemporal correlation modeling.
• The Adam optimizer is used to train the final layer of the AlexNet model, while the 1D-CNN is trained end-to-end on the multimodal signal sequences.
• The framework leverages the complementary nature of EEG (temporal resolution), fNIRS (hemodynamic response), and pupil diameter (autonomic arousal) to enhance workload detection robustness.

Experimental results

Research questions

• RQ1 Can a multimodal deep learning approach accurately detect cognitive workload during laparoscopic surgical tasks using EEG, fNIRS, and pupil diameter signals?
• RQ2 Does transfer learning with a pre-trained AlexNet improve detection performance for cognitive workload presence in surgical settings?
• RQ3 Can a 1D-CNN effectively classify different surgical task conditions based on fused physiological signals, reflecting varying levels of mental demand?
• RQ4 How do hemodynamic responses (HbO2 and HbR) from fNIRS correlate with subjective workload ratings and task conditions?
• RQ5 To what extent do auditory distractions and cognitive load tasks modulate physiological markers of cognitive workload in novice surgeons?

Key findings

• The proposed two-stage deep learning model achieved 100% accuracy in detecting the presence of cognitive workload using transfer learning with fine-tuned AlexNet.
• The second-stage 1D-CNN model achieved an overall classification accuracy of 93% in identifying the specific surgical task condition associated with elevated cognitive workload.
• fNIRS data revealed significant increases in HbO2 and HbR levels during task conditions 2 and 4—those involving auditory distractions and cognitive load—indicating heightened prefrontal cortex activation.
• Subjective workload ratings (SURG-TLX) confirmed a notable increase in mental demand during task 2 and 4, aligning with objective physiological changes.
• The combination of EEG, fNIRS, and pupil diameter signals enabled robust detection of workload changes, outperforming unimodal approaches by capturing complex spatiotemporal correlations.
• The framework demonstrated low computational cost and high accuracy, supporting its potential for real-time deployment in operating rooms.