[Paper Review] More than Model Documentation: Uncovering Teachers' Bespoke Information Needs for Informed Classroom Integration of ChatGPT

Mei Tan, Hariharan Subramonyam|arXiv (Cornell University)|Sep 25, 2023

Online Learning and AnalyticsComputer Science3 citations

TL;DR

This study identifies critical information gaps teachers face when integrating ChatGPT into classrooms, revealing that standard model documentation fails to meet their pedagogical and technical needs. The authors propose an interactive model documentation framework to bridge the gap between AI transparency and classroom practice, enabling teachers to effectively assess and apply ChatGPT’s capabilities with diverse learners.

ABSTRACT

ChatGPT has entered classrooms, but not via the typical route of other educational technology, which includes comprehensive training, documentation, and vetting. Consequently, teachers are urgently tasked to assess its capabilities to determine potential effects on student learning and instruct their use of ChatGPT. However, it is unclear what support teachers have and need and whether existing documentation, such as model cards, provides adequate direction for educators in this new paradigm. By interviewing 22 middle- and high-school teachers, we connect the discourse on AI transparency and documentation with educational technology integration, highlighting the critical information needs of teachers. Our findings reveal that teachers confront significant information gaps, lacking clarity on exploring ChatGPT's capabilities for bespoke learning tasks and ensuring its fit with the needs of diverse learners. As a solution, we propose a framework for interactive model documentation that empowers teachers to navigate the interplay between pedagogical and technical knowledge.

Motivation & Objective

• To investigate the specific information needs of K-12 teachers when evaluating and integrating ChatGPT into classroom instruction.
• To examine the limitations of existing AI documentation—such as model cards and educator guides—in supporting teachers’ practical decision-making.
• To identify the disconnect between technical AI transparency and the pedagogical realities of classroom implementation.
• To develop a framework that empowers teachers to navigate the interplay between technical capabilities and educational goals.
• To address ethical and equity concerns arising from uneven teacher access to AI literacy and support.

Proposed method

• Conducted semi-structured interviews with 22 experienced middle and high school teachers to explore their information-seeking behaviors and evaluation strategies for ChatGPT.
• Analyzed teachers’ approaches to testing ChatGPT’s capabilities, including their use of prompts and mental models for predicting outputs.
• Mapped observed information gaps to specific shortcomings in existing AI documentation, particularly in domain relevance and actionable guidance.
• Proposed a framework for interactive model documentation that integrates pedagogical context, technical affordances, and risk awareness for educators.
• Evaluated the framework’s potential to support teachers in assessing fairness, bias, and alignment with curricular goals.
• Used qualitative thematic analysis to identify recurring challenges in prompt design, student use management, and ethical oversight.

Experimental results

Research questions

• RQ1 What specific information needs do teachers have when evaluating ChatGPT for classroom use, beyond what is provided in standard model documentation?
• RQ2 How do teachers currently explore and test ChatGPT’s capabilities, and what cognitive and technical limitations hinder effective integration?
• RQ3 In what ways do existing AI transparency documents—such as model cards and educator guides—fail to support teachers in making informed pedagogical decisions?
• RQ4 What structural and design features are necessary for model documentation to effectively serve educators in the context of generative AI integration?
• RQ5 How can AI documentation be reimagined to address the unique needs of diverse learners and prevent unintended harms in educational settings?

Key findings

• Teachers lack clarity on how to explore ChatGPT’s capabilities for bespoke learning tasks, especially in alignment with specific curricular goals and diverse student needs.
• Many teachers approach prompt engineering with flawed mental models and narrow goals, resulting in inconsistent or low-quality outputs that limit their perception of the tool’s potential.
• Existing documentation, such as model cards and educator guides, fails to provide actionable, context-specific guidance for classroom implementation, leaving teachers to navigate technical and ethical complexities alone.
• Teachers express significant concerns about academic integrity, bias, and student safety, yet lack structured support to manage these risks or design effective AI-integrated assessments.
• The current model documentation is post-hoc and technical, deflecting responsibility to practitioners without offering tools to anticipate or mitigate context-dependent harms.
• There is a critical need for domain-specific, interactive documentation that connects technical affordances with pedagogical strategies and ethical considerations in real classroom contexts.