[Paper Review] n-Gage: Predicting in-class Emotional, Behavioural and Cognitive Engagement in the Wild

Nan Gao, Wei Shao+2 more|arXiv (Cornell University)|Jul 9, 2020

Flow Experience in Various Fields81 references32 citations

TL;DR

n-Gage predicts multidimensional in-class engagement (emotional, behavioural, cognitive) for high school students using wearable and environmental sensors in real-world classrooms, achieving MAE ~0.56–0.79 and RMSE ~0.72–0.98.

ABSTRACT

The study of student engagement has attracted growing interests to address problems such as low academic performance, disaffection, and high dropout rates. Existing approaches to measuring student engagement typically rely on survey-based instruments. While effective, those approaches are time-consuming and labour-intensive. Meanwhile, both the response rate and quality of the survey are usually poor. As an alternative, in this paper, we investigate whether we can infer and predict engagement at multiple dimensions, just using sensors. We hypothesize that multidimensional student engagement can be translated into physiological responses and activity changes during the class, and also be affected by the environmental changes. Therefore, we aim to explore the following questions: Can we measure the multiple dimensions of high school student's learning engagement including emotional, behavioural and cognitive engagement with sensing data in the wild? Can we derive the activity, physiological, and environmental factors contributing to the different dimensions of student engagement? If yes, which sensors are the most useful in differentiating each dimension of the engagement? Then, we conduct an in-situ study in a high school from 23 students and 6 teachers in 144 classes over 11 courses for 4 weeks. We present the n-Gage, a student engagement sensing system using a combination of sensors from wearables and environments to automatically detect student in-class multidimensional learning engagement. Experiment results show that n-Gage can accurately predict multidimensional student engagement in real-world scenarios with an average MAE of 0.788 and RMSE of 0.975 using all the sensors. We also show a set of interesting findings of how different factors (e.g., combinations of sensors, school subjects, CO2 level) affect each dimension of the student learning engagement.

Motivation & Objective

• Motivate the need for automated, sensor-based engagement measurement to supplement or replace time-consuming surveys.
• Investigate whether multidimensional engagement (emotional, behavioural, cognitive) can be inferred from physiological, activity, and environmental data collected in the wild.
• Identify which sensors most effectively differentiate each engagement dimension.
• Develop and validate a classroom sensing system (n-Gage) that fuses wearable and indoor environmental data for engagement prediction.

Proposed method

• Collect a large, diverse in-the-wild dataset from 23 students and 6 teachers across 144 classes over 4 weeks in a high school.
• Use Empatica E4 wristbands to capture EDA, PPG/HRV, accelerometry, and skin temperature, plus Netatmo indoor sensors for temperature and CO2.
• Gather self-report engagement via adapted In-class Student Engagement Questionnaires (ISEQ) for behavioural, emotional, and cognitive dimensions.
• Preprocess data with class-period segmentation (IGTS), artifact removal, decomposition of EDA, HRV estimation, and normalization.
• Extract features from physiological signals, activity, and environmental data, including skin temperature and indoor environment measures.
• Predict multidimensional engagement with LightGBM regressors, evaluating using MAE and RMSE across all sensors.

Experimental results

Research questions

• RQ1 Can we measure multiple dimensions of high school students’ learning engagement (emotional, behavioural, cognitive) using sensing data in the wild?
• RQ2 What activity, physiological, and environmental factors contribute to different engagement dimensions, and which sensors best differentiate them?
• RQ3 Which sensors and feature representations most improve accuracy of predicting each engagement dimension?
• RQ4 How do environmental factors (e.g., CO2) influence engagement dimensions in real classroom settings?

Key findings

• n-Gage can predict multidimensional engagement with MAE around 0.56 and RMSE around 0.72 when using all sensors.
• Across in-the-wild data, using all sensors yields MAE ≈ 0.788 and RMSE ≈ 0.975 for engagement prediction.
• Environmental factors like CO2 levels negatively affect cognitive engagement, highlighting ventilation relevance.
• The study provides evidence on the utility of combining physiological signals, movement, and indoor environmental data for engagement estimation in real classrooms.
• The dataset comprises 331 class sessions after cleaning, involving 23 students and 6 teachers across 105 analyzed classes.