[Paper Review] SPL: A Socratic Playground for Learning Powered by Large Language Model

Liang Zhang, Jionghao Lin+3 more|arXiv (Cornell University)|Jun 20, 2024

Innovative Teaching and Learning MethodsPsychology3 citations

TL;DR

This paper introduces SPL, a dialogue-based Intelligent Tutoring System powered by GPT-4 that employs the Socratic teaching method to foster critical thinking through interactive, multi-turn tutoring dialogues. Using advanced prompt engineering, SPL automates lesson design and delivers adaptive, personalized learning experiences, with pilot results showing improved learner engagement and critical thinking in essay writing tasks.

ABSTRACT

Dialogue-based Intelligent Tutoring Systems (ITSs) have significantly advanced adaptive and personalized learning by automating sophisticated human tutoring strategies within interactive dialogues. However, replicating the nuanced patterns of expert human communication remains a challenge in Natural Language Processing (NLP). Recent advancements in NLP, particularly Large Language Models (LLMs) such as OpenAI's GPT-4, offer promising solutions by providing human-like and context-aware responses based on extensive pre-trained knowledge. Motivated by the effectiveness of LLMs in various educational tasks (e.g., content creation and summarization, problem-solving, and automated feedback provision), our study introduces the Socratic Playground for Learning (SPL), a dialogue-based ITS powered by the GPT-4 model, which employs the Socratic teaching method to foster critical thinking among learners. Through extensive prompt engineering, SPL can generate specific learning scenarios and facilitates efficient multi-turn tutoring dialogues. The SPL system aims to enhance personalized and adaptive learning experiences tailored to individual needs, specifically focusing on improving critical thinking skills. Our pilot experimental results from essay writing tasks demonstrate SPL has the potential to improve tutoring interactions and further enhance dialogue-based ITS functionalities. Our study, exemplified by SPL, demonstrates how LLMs enhance dialogue-based ITSs and expand the accessibility and efficacy of educational technologies.

Motivation & Objective

• To develop a dialogue-based Intelligent Tutoring System (ITS) that emulates expert human tutoring through the Socratic method.
• To leverage large language models (LLMs), specifically GPT-4, to automate lesson design and enable adaptive, personalized learning experiences.
• To improve critical thinking and deep comprehension in learners through interactive, multi-turn dialogues guided by wh-questions (What?, Why?, How?, Who?, When?).
• To reduce reliance on manual lesson design and predefined rules by using LLMs for dynamic, context-aware tutoring.
• To evaluate the system’s effectiveness in enhancing learner engagement and educational outcomes in essay writing tasks.

Proposed method

• The system uses GPT-4 with advanced prompt engineering to generate context-aware, human-like tutoring responses.
• It applies the Socratic method by initiating dialogues with open-ended wh-questions to stimulate self-reflection and critical thinking.
• A standardized prompt template is used to automate lesson creation for specific learning scenarios and educational contexts.
• The interface integrates educational principles (e.g., Zone of Proximal Development) and supports multi-turn dialogues to guide learners step-by-step.
• Prompt strategies include chain-of-thought and few-shot prompting to enhance reasoning and response quality.
• The system is designed to adapt to diverse learner profiles and educational domains through flexible, generative AI capabilities.

Figure 1: An Example Interactive Dialogue Interface of SPL System.

Experimental results

Research questions

• RQ1 Can a GPT-4-powered dialogue system effectively simulate the Socratic teaching method to enhance learners’ critical thinking?
• RQ2 To what extent does SPL improve learner engagement and comprehension in essay writing tasks compared to traditional ITS approaches?
• RQ3 How well can prompt engineering enable automated, adaptive, and personalized tutoring without predefined rules?
• RQ4 What is the impact of multi-turn, wh-question-driven dialogues on learner performance and reflection?
• RQ5 How robust and consistent are the system’s feedback and evaluation capabilities in real educational scenarios?

Key findings

• Pilot testing demonstrated that SPL significantly enhances learner engagement and enjoyment in essay writing tasks through interactive, Socratic dialogues.
• The system successfully fosters critical thinking by guiding learners through structured, multi-turn dialogues using wh-questions such as 'Why?' and 'How?'
• SPL reduces dependency on manual lesson design and rule-based systems by automating pedagogical content generation via GPT-4.
• The system exhibits strong potential for scalability and adaptability across diverse educational contexts and learner profiles.
• Preliminary results indicate that SPL improves the quality of tutoring interactions and supports deeper comprehension in learners.
• The evaluation framework for essay submissions shows promise in assessing robustness, sensitivity, and psychometric consistency of AI-generated feedback.

Figure 2: The Socratic Playground for Learning System Architecture.