[Paper Review] Leveraging Crowd for Game-based Learning: A Case Study of Privacy Education Game Design and Evaluation by Crowdsourcing

Wendy Wang, Yu Tao+3 more|arXiv (Cornell University)|Mar 9, 2016

Mobile Crowdsensing and Crowdsourcing25 references3 citations

TL;DR

This paper proposes a crowdsourcing approach to design and evaluate a game-based privacy education system using non-expert workers from Amazon Mechanical Turk. By collecting game design ideas and evaluating the final game with both crowds and privacy experts, the study demonstrates that non-experts can generate high-quality, creative, and educationally effective game concepts, with crowd evaluations showing strong alignment (85% agreement) and even higher engagement than expert feedback.

ABSTRACT

As the Internet grows in importance, it is vital to develop methods and techniques for educating end-users to improve their awareness of online privacy. Web-based education tools have been proven effective in many domains and have been increasingly adopted by many online professional and educational services. However, the design and development of Web-based education tools for online privacy is still in the early stage. The traditional solutions always involve privacy experts who have sophisticated expertise. Such involvement can make the tool development costly. Furthermore, it is not clear how inspiring and effective these education tools are to general users of varying backgrounds, specially to novice users who have rarely dealt with online privacy issues before. In this paper, we design, develop, and evaluate a game-based privacy learning system by leveraging the wisdom of a crowd of non-experts on Amazon Mechanic Turk. Empirical study demonstrates that the crowd can provide high-quality ideas of designing and developing a practical, educational privacy learning game.

Motivation & Objective

• To explore whether non-expert crowds can contribute high-quality ideas for designing effective game-based privacy education tools.
• To evaluate the effectiveness of a privacy education game using both crowds and privacy experts, assessing usability and educational impact.
• To investigate the feasibility of using crowdsourcing for both creative design and performance evaluation in educational game development.
• To develop metrics for assessing the correctness and novelty of crowd-generated game design ideas.
• To compare crowd feedback with expert feedback in terms of quality, engagement, and educational value.

Proposed method

• Utilized Amazon Mechanical Turk to collect game design ideas from a crowd of non-experts through structured microtasks.
• Aggregated and synthesized crowd-sourced ideas into a coherent game design prototype for a privacy education game.
• Developed a three-phase system: idea collection, game development, and game evaluation.
• Applied multi-scale evaluation metrics (correctness, novelty, feasibility) to assess the quality of crowd-generated ideas.
• Conducted game evaluation with both the crowd and a group of privacy experts using Likert-scale and qualitative feedback.
• Used prediction voting and Likert-scale ratings to compare the effectiveness of different evaluation strategies.

Experimental results

Research questions

• RQ1 Can non-expert crowds generate high-quality, creative, and educationally effective ideas for a privacy education game?
• RQ2 How does the quality and engagement of crowd feedback compare to that of privacy experts in evaluating a game’s educational impact?
• RQ3 To what extent can crowd-sourced ideas be aggregated and transformed into a functional, usable game prototype?
• RQ4 What metrics best capture the correctness and novelty of crowd-generated educational game concepts?
• RQ5 Can crowds effectively evaluate the usability and educational value of a game, especially when compared to expert evaluations?

Key findings

• The crowd generated game design ideas with 85% agreement with experts on the overall quality, indicating strong consensus and reliability.
• Crowd evaluations showed 15% higher engagement than expert evaluations, suggesting greater motivational impact for general users.
• Experts rated the character Lucas as the most educational, while the crowd rated Aiden as the most inspiring, highlighting divergent perceptions of educational effectiveness.
• The study found that crowd-generated ideas were highly novel and practical, with significant integration of creative and functional features.
• The evaluation process revealed that crowd feedback was particularly effective in identifying engaging and intuitive game elements, even when experts considered them trivial.
• The results confirm that crowdsourcing can be a cost-effective and innovative method for both designing and evaluating educational games, especially when targeting diverse, non-expert audiences.