[Paper Review] Language Models as Science Tutors

Alexis Chevalier, Jiayi Geng+20 more|arXiv (Cornell University)|Feb 16, 2024

Innovative Teaching and Learning Methods5 citations

TL;DR

The paper introduces TutorEval, a long-context science QA benchmark, and TutorChat, a long-context dialogue dataset, to train and evaluate LM tutors for STEM education, showing that science-text fine-tuning and TutorChat greatly improve performance on TutorEval and math tasks.

ABSTRACT

NLP has recently made exciting progress toward training language models (LMs) with strong scientific problem-solving skills. However, model development has not focused on real-life use-cases of LMs for science, including applications in education that require processing long scientific documents. To address this, we introduce TutorEval and TutorChat. TutorEval is a diverse question-answering benchmark consisting of questions about long chapters from STEM textbooks, written by experts. TutorEval helps measure real-life usability of LMs as scientific assistants, and it is the first benchmark combining long contexts, free-form generation, and multi-disciplinary scientific knowledge. Moreover, we show that fine-tuning base models with existing dialogue datasets leads to poor performance on TutorEval. Therefore, we create TutorChat, a dataset of 80,000 long synthetic dialogues about textbooks. We use TutorChat to fine-tune Llemma models with 7B and 34B parameters. These LM tutors specialized in math have a 32K-token context window, and they excel at TutorEval while performing strongly on GSM8K and MATH. Our datasets build on open-source materials, and we release our models, data, and evaluations.

Motivation & Objective

• Motivate the need for real-life, long-context LM assistance in science education beyond short-context benchmarks.
• Create TutorEval, a long-context, expert-authored question set spanning multiple STEM domains to evaluate LM tutoring ability.
• Develop TutorChat, a large long-context dialogue dataset to fine-tune LM tutors on textbook-like interactions.
• Demonstrate that fine-tuning solely on dialogue data is insufficient; scientific-text and TutorChat data are crucial for strong TutorEval performance.
• Show that long-context models specialized in science and math can rival strong baselines on TutorEval, GSM8K, and MATH.

Proposed method

• Construct TutorEval with 834 questions across Math, CS, Physics, Environment, and Life sciences from textbook chapters (average ~1,800 words, up to 6,100).
• Annotate ground-truth key points for each question to guide LM evaluation (LM evaluator uses these key points).
• Use GPT-4 as evaluator to grade LM tutor outputs against ground-truth key points; assess correlation with human judgments.
• Create TutorChat by generating 78K long, synthetic dialogues about textbook chapters (80K dialogues after expansion), using GPT-3.5-Turbo and GPT-4-Turbo.
• Extend LM context to 32K tokens (Long-context) and fine-tune Llemma-7B-32K on TutorChat and MathMix datasets (TutorChat-STEM + MetaMath).
• Propose MathMix (TutorChat-STEM + MetaMath) to boost math capabilities while maintaining TutorEval performance.

Figure 1: Example from TutorEval . Given the chapter, the student asks a question to the LM Tutor. Both the chapter and the question are fed to the LM Tutor to generate the answer. GPT-4 assesses the generation by referencing the human annotated key points ( blue : the tutoring task; yellow : evalua

Experimental results

Research questions

• RQ1 How can LM tutors be effectively evaluated on long-context scientific tasks beyond final-answer correctness?
• RQ2 Does training on scientific text and long-context dialogue data improve TutorEval performance compared to base or dialogue-only fine-tuning?
• RQ3 What is the impact of combining TutorChat with math-focused data (MetaMath) on math problem solving vs. general science tutoring?
• RQ4 Can open-book vs. closed-book settings reveal strengths/limitations of long-context science tutors?
• RQ5 How different base models and data mixtures perform on TutorEval, GSM8K, and MATH?

Key findings

• TutorEval is a challenging long-context benchmark across multiple science domains requiring advanced scientific knowledge and processing of textbook content.
• GPT-4 correlates well with human judgments when used as an evaluator for TutorEval.
• Fine-tuning with scientific texts and TutorChat substantially improves TutorEval performance over base models and dialogue-only fine-tuning.
• Long-context models (32K tokens) trained on math and science data (MathMix) achieve strong math problem-solving performance while maintaining TutorEval competitiveness.
• A data mixture like MathMix (TutorChat-STEM + MetaMath) yields strong math capabilities (GSM8K/MATH) alongside solid TutorEval results, outperforming several baselines.
• TutorChat data quality (GPT-4 generated dialogues) can mitigate sycophancy and improve robustness to misleading questions; open-book dialogues outperform closed-book in many cases.

Figure 2: Left: performance breakdown on TutorEval by domains. Right: leaderboard of popular models on TutorEval . Our models, marked in bold, achieve competitive TutorEval performance.