[Paper Review] BloomNet: A Robust Transformer based model for Bloom's Learning Outcome Classification

Abdul Waheed, Muskan Goyal+4 more|arXiv (Cornell University)|Aug 16, 2021

Text and Document Classification Technologies6 citations

TL;DR

BloomNet is a novel transformer-based model that integrates linguistic and semantic features to classify course learning outcomes into six cognitive levels of Bloom’s Taxonomy. It achieves state-of-the-art performance and superior out-of-distribution generalization, outperforming strong baselines including RoBERTa and DistilRoBERTa, with a macro-F1 of 87.23% on in-distribution data and robustness to distribution shift.

ABSTRACT

Bloom taxonomy is a common paradigm for categorizing educational learning objectives into three learning levels: cognitive, affective, and psychomotor. For the optimization of educational programs, it is crucial to design course learning outcomes (CLOs) according to the different cognitive levels of Bloom Taxonomy. Usually, administrators of the institutions manually complete the tedious work of mapping CLOs and examination questions to Bloom taxonomy levels. To address this issue, we propose a transformer-based model named BloomNet that captures linguistic as well semantic information to classify the course learning outcomes (CLOs). We compare BloomNet with a diverse set of basic as well as strong baselines and we observe that our model performs better than all the experimented baselines. Further, we also test the generalization capability of BloomNet by evaluating it on different distributions which our model does not encounter during training and we observe that our model is less susceptible to distribution shift compared to the other considered models. We support our findings by performing extensive result analysis. In ablation study we observe that on explicitly encapsulating the linguistic information along with semantic information improves the model on IID (independent and identically distributed) performance as well as OOD (out-of-distribution) generalization capability.

Motivation & Objective

• To automate the classification of course learning outcomes (CLOs) into cognitive levels of Bloom’s Taxonomy, reducing manual effort in curriculum design.
• To improve model robustness by enhancing generalization to out-of-distribution (OOD) data, which is common in real-world educational NLP applications.
• To investigate the contribution of linguistic features—such as part-of-speech and named entity recognition—alongside semantic representations in improving classification performance.
• To evaluate the efficacy of state-of-the-art transformer models in the education domain, particularly for cognitive domain classification.
• To address the gap in multi-domain, cross-dataset generalization for Bloom’s taxonomy classification in educational NLP.

Proposed method

• BloomNet employs a RoBERTa-based language encoder to capture deep contextual semantics from CLO text.
• It incorporates a linguistic encapsulation module that processes part-of-speech (POS) and named entity recognition (NER) features to enrich syntactic and semantic input.
• A hierarchical word attention mechanism is applied to focus on key words and phrases relevant to cognitive level classification.
• The model is fine-tuned end-to-end on two datasets with 600 and 141 samples respectively, using cross-validation to ensure stability.
• Model robustness is evaluated via OOD generalization tests on data distributions not seen during training.
• An ablation study isolates the contributions of word attention and linguistic encapsulation to assess their impact on both in-distribution and out-of-distribution performance.

Experimental results

Research questions

• RQ1 Can a transformer-based model effectively classify course learning outcomes into the six cognitive levels of Bloom’s Taxonomy?
• RQ2 How does the integration of linguistic features (POS, NER) improve classification accuracy and OOD generalization compared to pure semantic modeling?
• RQ3 Do pretrained transformer models like RoBERTa generalize well to out-of-distribution data in the context of educational outcome classification?
• RQ4 What is the relative contribution of word-level attention and linguistic feature encapsulation to the model’s performance and robustness?
• RQ5 How does BloomNet compare in performance and generalization to strong baselines such as CNNs, LSTMs, and other transformer models?

Key findings

• BloomNet achieves a macro-F1 score of 87.23% on in-distribution (IID) data, significantly outperforming RoBERTa (81.67%) and other baselines.
• The ablation study confirms that combining word attention with linguistic encapsulation yields the highest performance, with a 5.56 percentage point improvement in macro-F1 over RoBERTa.
• Models with better IID performance, including BloomNet and pretrained transformers, also show stronger OOD generalization, though all pretrained models suffer a ~20% performance drop on OOD data.
• The linguistic encapsulation module improves OOD generalization, suggesting that syntactic and named entity features help the model generalize beyond superficial statistical cues.
• Word attention alone improves performance by ~2.1% in macro-F1 over RoBERTa, indicating that focusing on key lexical elements enhances classification.
• Despite small dataset sizes (600 and 141 samples), BloomNet maintains consistent performance across cross-validation, indicating stability under limited data.