[Paper Review] EduChat: A Large-Scale Language Model-based Chatbot System for Intelligent Education

Yuhao Dan, Zhikai Lei+14 more|arXiv (Cornell University)|Aug 5, 2023

AI in Service InteractionsComputer Science34 citations

TL;DR

EduChat is an open-source LLM-based educational chatbot that combines domain-specific pre-training, fine-tuning on education-focused instructions, and retrieval-augmented open QA to improve open questions, essay assessment, Socratic teaching, and emotional support.

ABSTRACT

EduChat (https://www.educhat.top/) is a large-scale language model (LLM)-based chatbot system in the education domain. Its goal is to support personalized, fair, and compassionate intelligent education, serving teachers, students, and parents. Guided by theories from psychology and education, it further strengthens educational functions such as open question answering, essay assessment, Socratic teaching, and emotional support based on the existing basic LLMs. Particularly, we learn domain-specific knowledge by pre-training on the educational corpus and stimulate various skills with tool use by fine-tuning on designed system prompts and instructions. Currently, EduChat is available online as an open-source project, with its code, data, and model parameters available on platforms (e.g., GitHub https://github.com/icalk-nlp/EduChat, Hugging Face https://huggingface.co/ecnu-icalk ). We also prepare a demonstration of its capabilities online (https://vimeo.com/851004454). This initiative aims to promote research and applications of LLMs for intelligent education.

Motivation & Objective

• Explore how psychology and education theories can adapt general LLMs to education domains.
• Pre-train on a large educational corpus to learn domain knowledge.
• Fine-tune with task-specific instructions to activate education-specific functions.
• Incorporate retrieval to keep knowledge up-to-date and reduce hallucinations.
• Release EduChat as an open-source system to accelerate education-focused AI research.

Proposed method

• Pre-train on educational books and questions plus poetry and psychology texts to encode domain knowledge.
• Fine-tune on 500k high-quality customized instructions to activate educational functions.
• Use retrieval-augmented open QA to access up-to-date information and self-check for relevance.
• Design diverse system prompts to control tool usage and enable skills like Socratic teaching and emotional support.
• Translate and curate data for emotional support dialogues (ESConv-zh) and develop fine-grained essay assessment datasets.
• Manual data cleaning and semantic deduplication using sentence-transformers to ensure data quality.

Figure 1: Distribution of educational data.

Experimental results

Research questions

• RQ1 How can EduChat align LLMs with educational abilities like essay assessment and Socratic teaching?
• RQ2 Does retrieval-augmented QA improve accuracy and reduce hallucinations in educational contexts?
• RQ3 Can psychology-based emotional support be effectively provided by fine-tuned LLMs in education?
• RQ4 What is the impact of specialized educational fine-tuning on performance compared to general LLMs of similar size?

Key findings

• EduChat (with retrieval) outperforms similar-sized baselines on C-Eval Chinese evaluation across disciplines.
• EduChat with retrieval achieves higher average scores than EduChat without retrieval (49.3 vs 40.7 in the reported metrics).
• Retrieval-augmented QA significantly improves performance on multi-discipline benchmarks and reduces fabrication risk.
• Fine-grained essay assessment and Socratic teaching capabilities are enhanced through targeted educational data fine-tuning.
• Psychology-based emotional support enables EduChat to act as a compassionate counselor in education contexts.

Figure 2: The overall framework of EduChat .