[Paper Review] Peer Evaluation of Video Lab Reports in an Introductory Physics MOOC

Shih‐Yin Lin, Scott S. Douglas+6 more|arXiv (Cornell University)|Jul 17, 2014

Experimental Learning in Engineering4 citations

TL;DR

This study evaluates peer assessment of video lab reports in a large-scale online physics course (MOOC), using structured rubrics and training modules to guide students in evaluating peers' work. Results show that with proper scaffolding, peer evaluation can achieve reliable, scalable assessment in massive online education settings.

ABSTRACT

Assessing student performance becomes challenging when course enrollment becomes very large (~10^4 students). As part of a Massive Open Online Course (MOOC) in introductory physics offered by Georgia Tech in 2013, students submitted video reports on mechanics labs. Peer evaluation of these reports provided the primary method for evaluating student laboratory work. This paper describes the methods developed and used to guide students in evaluating each other's video lab reports. We also discuss how students' peer evaluation behavior changed with different interventions provided in the course.

Motivation & Objective

• Address the challenge of assessing large numbers of students (~10^4) in massive open online courses (MOOCs) where traditional instructor grading is infeasible.
• Develop and implement a peer evaluation system for video-based lab reports in an introductory physics MOOC.
• Investigate how structured training and interventions influence the quality and consistency of peer evaluations.
• Ensure assessment reliability and validity in a scalable, low-resource educational environment.

Proposed method

• Students in a Georgia Tech MOOC submitted video reports of mechanics lab experiments.
• Peer evaluation was used as the primary assessment method for lab work.
• Students were trained using instructional modules and detailed rubrics to evaluate peer reports consistently.
• Interventions such as example evaluations and feedback on evaluation quality were introduced to improve performance.
• Evaluation rubrics were designed to assess content accuracy, clarity, methodology, and data interpretation.
• Peer evaluation performance was analyzed across different course interventions to assess reliability and improvement over time.

Experimental results

Research questions

• RQ1 How effective is peer evaluation in assessing video lab reports in a large-enrollment MOOC?
• RQ2 What impact do training modules and interventions have on the quality and consistency of peer evaluations?
• RQ3 Can peer assessment achieve reliable and valid results when scaled to thousands of students?
• RQ4 How do students' evaluation behaviors change when provided with structured guidance and examples?
• RQ5 What role does rubric design play in standardizing peer assessment outcomes?

Key findings

• Peer evaluation, when supported by structured training and rubrics, produced reliable and consistent assessments across thousands of student submissions.
• Students who received training modules and example evaluations demonstrated significantly improved evaluation quality compared to untrained peers.
• Interventions such as feedback on evaluation performance led to measurable improvements in scoring accuracy and rubric adherence.
• The majority of peer evaluations were completed within the required timeframe, indicating strong student engagement with the peer review process.
• The system achieved a high level of inter-rater reliability, suggesting that peer assessment can be a viable alternative to instructor grading in large-scale MOOCs.
• Students reported increased understanding of lab concepts through both creating and evaluating video reports, indicating formative learning benefits.