[Paper Review] Robosourcing Educational Resources -- Leveraging Large Language Models for Learnersourcing

Paul Denny, Sami Sarsa+2 more|arXiv (Cornell University)|Nov 9, 2022

Natural Language Processing TechniquesComputer Science18 citations

TL;DR

This paper introduces robosourcing—a hybrid approach that leverages large language models (LLMs) like OpenAI Codex to generate educational programming exercises, with learners providing input priming and evaluating outputs. It demonstrates that ~33% of LLM-generated exercises are directly usable for teaching, significantly reducing human effort while maintaining quality, thus addressing motivation and quality challenges in learnersourcing.

ABSTRACT

In this article, we introduce and evaluate the concept of robosourcing for creating educational content. Robosourcing lies in the intersection of crowdsourcing and large language models, where instead of a crowd of humans, requests to large language models replace some of the work traditionally performed by the crowd. Robosourcing includes a human-in-the-loop to provide priming (input) as well as to evaluate and potentially adjust the generated artefacts; these evaluations could also be used to improve the large language models. We propose a system to outline the robosourcing process. We further study the feasibility of robosourcing in the context of education by conducting an evaluation of robosourced and programming exercises, generated using OpenAI Codex. Our results suggest that robosourcing could significantly reduce human effort in creating diverse educational content while maintaining quality similar to human-created content.

Motivation & Objective

• To address low learner motivation and inconsistent content quality in learnersourcing by integrating large language models (LLMs) into the process.
• To evaluate whether LLM-generated educational content can serve as a viable starting point for learners, reducing the burden of content creation.
• To investigate the feasibility of using LLMs as scaffolding tools in learnersourcing, shifting learners’ roles from creators to evaluators.
• To assess the quality, novelty, and usability of LLM-generated programming exercises in an educational context.
• To explore the potential for feedback from human evaluations to improve LLMs for educational content generation.

Proposed method

• Leveraged OpenAI Codex as the LLM to generate programming exercises based on natural language prompts provided by learners.
• Used structured priming (e.g., topic, concepts, task description) to guide LLMs in generating domain-specific exercises.
• Employed a human-in-the-loop model where learners evaluated generated exercises for correctness, clarity, and usability using predefined criteria.
• Applied automated testing to validate functional correctness of generated code, ensuring reliability.
• Evaluated exercises for novelty using Google search to confirm they were not copied from existing sources.
• Collected feedback on content quality and potential biases in exercise descriptions.

Experimental results

Research questions

• RQ1 Can LLMs like OpenAI Codex generate programming exercises of sufficient quality to be used in educational settings?
• RQ2 To what extent do learners find robosourced exercises suitable for evaluation and modification?
• RQ3 How does the quality and novelty of LLM-generated exercises compare to human-created content?
• RQ4 What impact does the use of LLMs have on learner motivation and engagement in content creation tasks?
• RQ5 Can human evaluations of LLM-generated content contribute to improving the LLMs themselves?

Key findings

• Approximately one-third (33%) of the LLM-generated programming exercises were directly usable for teaching, meeting all evaluation criteria.
• Over 90% of the generated exercises were novel, as confirmed by Google search, indicating minimal duplication with existing content.
• The exercises were diverse in theme and concept, with themes like 'music' and concepts like 'class', 'list', and 'conditional' easily adjustable via prompt engineering.
• Learners were able to evaluate and modify the exercises efficiently, suggesting a shift from content creation to content curation is feasible and effective.
• No offensive content or personally identifiable information was detected in the generated exercises, though gender bias was observed in problem descriptions (e.g., male names more common).
• Feedback from human evaluations could potentially be used to fine-tune LLMs for better performance in educational content generation.