[论文解读] Integrating A.I. in Higher Education: Protocol for a Pilot Study with 'SAMCares: An Adaptive Learning Hub'

Syed Hasib Akhter Faruqui, Nazia Tasnim+3 more|arXiv (Cornell University)|May 1, 2024

Innovative Teaching and Learning Methods被引用 6

一句话总结

本论文协议描述了一项为期一年的随机对照试验，评估 SAMCares，基于 LLM 的自适应学习中心，使用 Retriever-Augmented Generation (RAG) 来为 SHSU 学生提供具情境感知的 AI 辅导。它概述了研究设计、数据收集和开源发布计划。

ABSTRACT

Learning never ends, and there is no age limit to grow yourself. However, the educational landscape may face challenges in effectively catering to students' inclusion and diverse learning needs. These students should have access to state-of-the-art methods for lecture delivery, online resources, and technology needs. However, with all the diverse learning sources, it becomes harder for students to comprehend a large amount of knowledge in a short period of time. Traditional assistive technologies and learning aids often lack the dynamic adaptability required for individualized education plans. Large Language Models (LLM) have been used in language translation, text summarization, and content generation applications. With rapid growth in AI over the past years, AI-powered chatbots and virtual assistants have been developed. This research aims to bridge this gap by introducing an innovative study buddy we will be calling the 'SAMCares'. The system leverages a Large Language Model (LLM) (in our case, LLaMa-2 70B as the base model) and Retriever-Augmented Generation (RAG) to offer real-time, context-aware, and adaptive educational support. The context of the model will be limited to the knowledge base of Sam Houston State University (SHSU) course notes. The LLM component enables a chat-like environment to interact with it to meet the unique learning requirements of each student. For this, we will build a custom web-based GUI. At the same time, RAG enhances real-time information retrieval and text generation, in turn providing more accurate and context-specific assistance. An option to upload additional study materials in the web GUI is added in case additional knowledge support is required. The system's efficacy will be evaluated through controlled trials and iterative feedback mechanisms.

研究动机与目标

• Investigate whether GenAI tools can enhance learning experiences and academic performance in higher education.
• Assess knowledge gains, student satisfaction, and cognitive load when using SAMCares versus traditional materials.
• Develop and evaluate an adaptive, context-aware AI tutoring system grounded in SHSU course notes.

提出的方法

• Use LLaMa-2 70B as the base model with a fixed SHSU course notes knowledgebase.
• Implement a Retriever-Augmented Generation (RAG) system to provide context-aware responses.
• Provide a web-based GUI with options to upload additional study materials and secure SHSU credentialed access.
• Conduct a year-long stratified randomized controlled trial with 150 freshmen from SHSU Engineering Technology.
• Collect quantitative data (exam scores, topic assessments) and qualitative data (pre-/post-surveys, interviews, eye-tracking).
• Perform power analysis to justify sample size (80% power, alpha 0.05) and analyze learning gains and satisfaction as primary/secondary outcomes.

Figure 1 : Schematic Representation of Randomization and Data Collection Process for SAMCares Tool Evaluation

实验结果

研究问题

• RQ1 Does access to SAMCares improve test scores and topic assessments compared with traditional materials?
• RQ2 What are the effects of SAMCares on student satisfaction and perceived cognitive load during learning?
• RQ3 How do eye-tracking metrics (engagement and fatigue) differ between SAMCares users and non-users?
• RQ4 What qualitative themes emerge regarding usability, perceived usefulness, and equity in AI-assisted learning?

主要发现

• No empirical results are reported in this protocol paper; the study aims to determine whether SAMCares improves learning outcomes and satisfaction.
• The central hypothesis is that SAMCares integration will yield greater knowledge gains, higher student satisfaction, and lower cognitive load than traditional methods.
• The protocol emphasizes rigorous data collection (exams, surveys, eye-tracking, interviews) and plans for de-identified data sharing and open-source release of SAMCares.
• The study acknowledges limitations such as single-site scope, potential misuse, infrastructure needs, and accessibility considerations for students with disabilities.

Figure 2 : Retriever-Augmented Generation (RAG) system for SAMCares : Process flowchart for generating context-aware responses using vector embeddings and LLAMA 2 70B model.