[Paper Review] Lessons Learnt from a Multimodal Learning Analytics Deployment In-the-wild

Roberto Martínez‐Maldonado, Vanessa Echeverría+11 more|arXiv (Cornell University)|Mar 16, 2023

Online and Blended Learning4 citations

TL;DR

This paper presents actionable lessons from a 2-year in-the-wild multimodal learning analytics (MMLA) deployment with 399 students and 17 educators, using video, audio, physiological sensors, and indoor positioning. It identifies critical challenges in technology integration, data privacy, ethical consent, interface design, and sustainability, advocating for human-centred, iterative design to support ethical and effective MMLA in authentic educational settings.

ABSTRACT

Multimodal Learning Analytics (MMLA) innovations make use of rapidly evolving sensing and artificial intelligence algorithms to collect rich data about learning activities that unfold in physical learning spaces. The analysis of these data is opening exciting new avenues for both studying and supporting learning. Yet, practical and logistical challenges commonly appear while deploying MMLA innovations "in-the-wild". These can span from technical issues related to enhancing the learning space with sensing capabilities, to the increased complexity of teachers' tasks and informed consent. These practicalities have been rarely discussed. This paper addresses this gap by presenting a set of lessons learnt from a 2-year human-centred MMLA in-the-wild study conducted with 399 students and 17 educators. The lessons learnt were synthesised into topics related to i) technological/physical aspects of the deployment; ii) multimodal data and interfaces; iii) the design process; iv) participation, ethics and privacy; and v) the sustainability of the deployment.

Motivation & Objective

• To address the lack of practical insights into deploying multimodal learning analytics (MMLA) in authentic educational settings.
• To identify and document real-world challenges in technology integration, data collection, and ethical considerations during MMLA deployments.
• To support the sustainable and ethical adoption of MMLA by educators and learners through evidence-based design principles.
• To bridge the gap between prototype MMLA systems and scalable, usable implementations in real classrooms.
• To inform future MMLA research and practice by synthesizing lessons from a longitudinal, human-centred deployment in authentic learning environments.

Proposed method

• Conducted a 2-year in-the-wild MMLA study across multiple authentic learning settings with 399 students and 17 educators.
• Deployed multimodal sensors including video, audio, physiological wristbands, and indoor positioning systems to capture rich learning data.
• Used iterative design with two study iterations, adapting to teacher and student feedback and evolving educational activities.
• Collected data through interviews, observations, and system logs to assess perceptions, intrusiveness, and usability.
• Synthesized lessons into five thematic areas: technological/physical setup, multimodal data and interfaces, design process, ethics and privacy, and sustainability.
• Applied human-centred design principles to co-create solutions with stakeholders, ensuring alignment with educational practices and ethical standards.

Experimental results

Research questions

• RQ1 What practical challenges emerge when deploying multimodal learning analytics in authentic, real-world educational environments?
• RQ2 How do students and educators perceive the intrusiveness and ethical implications of sensor-based data collection in learning spaces?
• RQ3 What design strategies improve the usability and sustainability of MMLA systems in real classrooms?
• RQ4 How can multimodal data interfaces be designed to respect privacy while supporting meaningful reflection and learning?
• RQ5 What factors influence the long-term adoption and scalability of MMLA technologies in educational institutions?

Key findings

• Sensor deployment in real classrooms introduced significant technical and logistical challenges, including complex installation, calibration, and maintenance requirements.
• Students and educators expressed concerns about privacy and data sensitivity, particularly when multimodal data were visualized in shared interfaces.
• Ethical consent processes were complex; participants often lacked full understanding of data usage, especially regarding long-term storage and sharing.
• The design of data interfaces significantly influenced user trust and willingness to participate, with visible data displays increasing perceived surveillance.
• Sustainability was undermined by high maintenance demands and lack of institutional support, even when initial engagement was strong.
• Teachers’ workload increased due to the need to manage new data flows and integrate MMLA insights into existing pedagogical practices.