[Paper Review] Continuous Mental Effort Evaluation during 3D Object Manipulation Tasks based on Brain and Physiological Signals

Dennis Wobrock, Jérémy Frey+4 more|arXiv (Cornell University)|May 29, 2015

Gaze Tracking and Assistive Technology29 references4 citations

TL;DR

This paper proposes a continuous, objective evaluation of mental effort during 3D object manipulation using EEG, ECG, and GSR signals. It demonstrates that these physiological signals can reliably estimate mental workload in real time, enabling fine-grained assessment of 3D user interface usability without interrupting users.

ABSTRACT

Designing 3D User Interfaces (UI) requires adequate evaluation tools to ensure good usability and user experience. While many evaluation tools are already available and widely used, existing approaches generally cannot provide continuous and objective measures of usa-bility qualities during interaction without interrupting the user. In this paper, we propose to use brain (with ElectroEncephaloGraphy) and physiological (ElectroCardioGraphy, Galvanic Skin Response) signals to continuously assess the mental effort made by the user to perform 3D object manipulation tasks. We first show how this mental effort (a.k.a., mental workload) can be estimated from such signals, and then measure it on 8 participants during an actual 3D object manipulation task with an input device known as the CubTile. Our results suggest that monitoring workload enables us to continuously assess the 3DUI and/or interaction technique ease-of-use. Overall, this suggests that this new measure could become a useful addition to the repertoire of available evaluation tools, enabling a finer grain assessment of the ergonomic qualities of a given 3D user interface.

Motivation & Objective

• To develop a continuous, non-intrusive method for assessing mental workload during 3D object manipulation tasks.
• To evaluate the feasibility of using brain and physiological signals as objective indicators of mental effort in real-time interaction.
• To provide a complementary evaluation tool to existing usability methods that require task interruption.
• To assess the ergonomic quality of 3D user interfaces and interaction techniques using physiological workload measures.
• To validate the approach with empirical data from participants performing 3D manipulation tasks using the CubTile device.

Proposed method

• Employed electroencephalography (EEG) to capture brain activity during 3D object manipulation tasks.
• Collected electrocardiography (ECG) and galvanic skin response (GSR) signals to assess autonomic nervous system activity related to mental workload.
• Used a regression-based approach to map physiological signals to continuous mental effort estimates.
• Conducted experiments with 8 participants using the CubTile as the 3D input device to perform object manipulation tasks.
• Applied signal preprocessing and feature extraction techniques to physiological data for robust workload estimation.
• Validated the mental effort estimation model using correlation analysis between physiological signals and subjective workload ratings.

Experimental results

Research questions

• RQ1 Can mental effort during 3D object manipulation be continuously estimated using EEG, ECG, and GSR signals?
• RQ2 How well do physiological signals correlate with subjective workload ratings during 3D interaction tasks?
• RQ3 To what extent can physiological-based workload estimation serve as a reliable alternative to traditional usability evaluation methods?
• RQ4 Can this approach detect variations in workload across different phases or difficulty levels of a 3D manipulation task?
• RQ5 Is the proposed method sensitive enough to detect differences in interaction technique usability in real time?

Key findings

• The proposed method successfully estimated mental effort in real time using EEG, ECG, and GSR signals during 3D object manipulation.
• Significant correlations were found between the estimated mental workload and subjective workload ratings, validating the approach.
• Physiological signals provided continuous workload monitoring without requiring user interruption or post-task questionnaires.
• The system detected variations in workload across different task phases, indicating sensitivity to task difficulty changes.
• The results suggest that this method can serve as a fine-grained, objective supplement to traditional usability evaluation in 3D UI research.
• The use of multimodal physiological signals improved the robustness and accuracy of workload estimation compared to single-signal approaches.