[Paper Review] The Elements of Item Response Theory and its Framework in Analyzing Introductory Astronomy College Student Misconceptions. I. Galaxies

Andrej Favia, Neil F. Comins+1 more|arXiv (Cornell University)|Jun 11, 2012

School Choice and Performance16 references3 citations

TL;DR

This study applies item response theory (IRT) to identify and rank the difficulty of astronomy misconceptions about galaxies among college students. Using factor analysis and IRT scoring, it reveals that spatial distribution misconceptions are the most persistent, with distinct optimal teaching sequences for adults (Factor 1, then 2, then 3) and younger learners (Factor 2, then 1, then 3), offering a data-driven framework for curriculum design in introductory astronomy.

ABSTRACT

This is the first in a series of papers that analyze college student beliefs in realms where common astronomy misconceptions are prevalent. Data was collected through administration of an inventory distributed at the end of an introductory college astronomy course. In this paper, we present the basic mathematics of item response theory (IRT), and then we use it to explore concepts related to galaxies. We show how IRT determines the difficulty of each galaxy topic under consideration. We find that the concept of galaxy spatial distribution presents the greatest challenge to students of all the galaxy topics. We also find and present the most logical sequence to teach galaxy topics as a function of the audience's age.

Motivation & Objective

• To quantify the persistence of galaxy-related misconceptions among college students using psychometric methods.
• To determine the relative difficulty of unlearning specific misconceptions about galaxies using item response theory (IRT).
• To identify the most effective sequence for teaching galaxy topics based on student age and misconception persistence.
• To provide a data-driven framework for improving astronomy instruction by targeting the most persistent misconceptions.
• To inform curriculum design by linking misconception difficulty to developmental age groups.

Proposed method

• Recode student responses from the AST 109 Misconceptions Inventory into 'misconception scores' reflecting increasing retention of incorrect beliefs.
• Apply principal components analysis (PCA) to group galaxy-related items based on correlated student responses, identifying three distinct misconception factors.
• Use the graded response model in item response theory (IRT) to estimate item difficulty and model the probability of endorsing each misconception level.
• Locate the intersection points of category response curves to determine the relative age at which misconceptions are dispelled, serving as a proxy for conceptual development.
• Sequence items within each factor by the location of intersection points, reflecting increasing misconception persistence.
• Compute a weighted average of intersection points across age groups to rank the overall teaching order for each factor.

Experimental results

Research questions

• RQ1 Which galaxy misconceptions are the most persistent among college students?
• RQ2 How does the difficulty of unlearning galaxy misconceptions vary across different age groups?
• RQ3 What is the most logical sequence for teaching galaxy topics to maximize conceptual understanding?
• RQ4 How can item response theory be used to quantify the persistence of misconceptions in astronomy education?
• RQ5 What is the relationship between spatial distribution misconceptions and other related misconceptions, such as those about the Milky Way's centrality?

Key findings

• The concept of galaxy spatial distribution is the most difficult for students to learn, with the lowest misconception factor coordinate (MFC) of -1.36, indicating the highest persistence.
• The misconception that there is only one galaxy is the easiest to dispel, reflecting the lowest difficulty in the IRT model.
• For adults, the optimal teaching sequence is Factor 1 (visual galaxy properties), followed by Factor 2 (centralization), then Factor 3 (spatial distribution).
• For children and adolescents, the most logical sequence is Factor 2 (centralization), then Factor 1 (visual galaxy properties), then Factor 3 (spatial distribution).
• The misconception that galaxies are randomly distributed is strongly correlated with the belief that we can see all stars in the Milky Way, indicating a shared underlying framework of spatial distribution.
• Students who believe the Milky Way is at the center of the universe or that we are at its center are also more likely to hold the misconception that only a few galaxies exist.