[Paper Review] An Iterative Transfer Learning Based Ensemble Technique for Automatic Short Answer Grading

Shourya Roy, Himanshu S. Bhatt+1 more|arXiv (Cornell University)|Sep 16, 2016

Topic Modeling36 references13 citations

TL;DR

This paper proposes an iterative transfer learning-based ensemble model for automatic short answer grading (ASAG) that combines a text classifier (using TF-IDF) and a numeric similarity classifier (based on similarity metrics to model answers). By leveraging canonical correlation analysis for transfer learning, the method achieves state-of-the-art performance on the SemEval-2013 SciEdu dataset without requiring labeled data for target questions, outperforming all winning supervised entries.

ABSTRACT

Automatic short answer grading (ASAG) techniques are designed to automatically assess short answers to questions in natural language, having a length of a few words to a few sentences. Supervised ASAG techniques have been demonstrated to be effective but suffer from a couple of key practical limitations. They are greatly reliant on instructor provided model answers and need labeled training data in the form of graded student answers for every assessment task. To overcome these, in this paper, we introduce an ASAG technique with two novel features. We propose an iterative technique on an ensemble of (a) a text classifier of student answers and (b) a classifier using numeric features derived from various similarity measures with respect to model answers. Second, we employ canonical correlation analysis based transfer learning on a common feature representation to build the classifier ensemble for questions having no labelled data. The proposed technique handsomely beats all winning supervised entries on the SCIENTSBANK dataset from the Student Response Analysis task of SemEval 2013. Additionally, we demonstrate generalizability and benefits of the proposed technique through evaluation on multiple ASAG datasets from different subject topics and standards.

Motivation & Objective

• To address the practical limitations of supervised ASAG methods that require labeled training data and model answers for every assessment task.
• To develop a robust, data-efficient ASAG technique that generalizes across diverse subject topics and grading schemes.
• To enable effective automatic grading of short answers even when no labeled student responses are available for the target question.
• To explore the feasibility and effectiveness of transfer learning in supervised ASAG to reduce dependency on extensive labeled data.

Proposed method

• Combines two complementary classifiers: a TF-IDF based text classifier trained on student answers and a numeric classifier using similarity metrics (e.g., F1, cosine, Lesk) between student and model answers.
• Employs canonical correlation analysis (CCA) to learn a shared low-dimensional feature space between source and target questions, enabling transfer learning.
• Uses an iterative refinement process to update the ensemble classifier by reweighting predictions based on consistency across both classifiers.
• Applies transfer learning by initializing the target question’s classifier using knowledge from 20 other questions in the dataset, even when no labeled data exists for the target.
• Employs a weighted ensemble strategy where predictions from both classifiers are combined using learned weights to improve robustness.
• Validates performance using leave-one-out cross-validation and evaluates generalization across multiple datasets and grading granularities.

Experimental results

Research questions

• RQ1 Can an ensemble of text and numeric similarity classifiers improve automatic short answer grading beyond single-model approaches?
• RQ2 How effective is canonical correlation analysis-based transfer learning in enabling zero-shot or few-shot ASAG for new questions without labeled data?
• RQ3 Does the iterative refinement process enhance model convergence and prediction accuracy across diverse question types and subject domains?
• RQ4 How does grading scheme granularity (e.g., 2-class vs. 11-class) affect the performance of the proposed transfer learning-based ASAG framework?

Key findings

• The proposed method outperformed all winning supervised entries on the SemEval-2013 SciEdu dataset, achieving the best performance in automatic short answer grading.
• On the CSD dataset, the method achieved a mean absolute error (MAE) of 1.07 using logistic regression as the base classifier, significantly outperforming LDA (MAE=1.63) and AdaBoost (MAE=1.58).
• The iterative algorithm converged in 5–6 iterations for most questions, indicating stable and efficient learning dynamics.
• Performance improved with coarser grading schemes: MAE decreased from 11-class (0.5-step) to 2-class (correct/incorrect) labeling, suggesting better generalization under sparse data.
• The framework generalizes well across different subject topics and standards, as demonstrated by evaluation on multiple ASAG datasets beyond the benchmark.
• The method remains effective even with minimal labeled data, demonstrating strong transfer learning capability through CCA-based feature alignment.