[Paper Review] Neural Skill Transfer from Supervised Language Tasks to Reading Comprehension

Todor Mihaylov, Zornitsa Kozareva+1 more|arXiv (Cornell University)|Nov 10, 2017

Topic Modeling34 references5 citations

TL;DR

This paper proposes a neural skill transfer approach that enhances reading comprehension by injecting knowledge from pre-trained, supervised language tasks—such as named entity recognition, textual entailment, paraphrase detection, and question type classification—into a simple question-answering model. The method achieves significant performance gains, especially in early training and with limited data, by leveraging token-wise label supervision and fine-tuning of skill encoders, outperforming baseline models without skill transfer.

ABSTRACT

Reading comprehension is a challenging task in natural language processing and requires a set of skills to be solved. While current approaches focus on solving the task as a whole, in this paper, we propose to use a neural network `skill' transfer approach. We transfer knowledge from several lower-level language tasks (skills) including textual entailment, named entity recognition, paraphrase detection and question type classification into the reading comprehension model. We conduct an empirical evaluation and show that transferring language skill knowledge leads to significant improvements for the task with much fewer steps compared to the baseline model. We also show that the skill transfer approach is effective even with small amounts of training data. Another finding of this work is that using token-wise deep label supervision for text classification improves the performance of transfer learning.

Motivation & Objective

• To improve reading comprehension performance by transferring knowledge from lower-level, supervised language tasks instead of training end-to-end.
• To investigate whether pre-trained 'skills' such as NER, textual entailment, and question type classification enhance a simple QA model.
• To evaluate the impact of skill transfer on training efficiency and data efficiency, particularly with small training subsets (2%, 5%, 10%, 25%).
• To examine the effectiveness of token-wise label supervision versus sentence-wise supervision in transfer learning for question answering.
• To provide a modular, interpretable framework for analyzing which specific language skills contribute to reading comprehension.

Proposed method

• Train separate Bi-LSTM encoders on labeled datasets for four language skills: NER (CoNLL 2012), textual entailment (SNLI), paraphrase detection (PPDB), and question type classification (TREC).
• Reuse the trained context encoder weights from each skill task to initialize the context encoder in a simple reading comprehension model.
• Integrate skill representations by sharing the context encoder weights across the QA model, enabling knowledge transfer without retraining the entire architecture.
• Apply token-wise label supervision for text classification tasks, where each token’s representation is projected to label space and summed to produce a sentence-level prediction.
• Fine-tune the skill encoder weights during QA training to adapt them to the target task, while also evaluating performance with frozen weights.
• Evaluate the model on SQuAD dev set using F1 and Exact Match (EM) metrics, comparing full-data and reduced-data training regimes.

Experimental results

Research questions

• RQ1 Does transferring knowledge from multiple supervised language tasks improve reading comprehension performance compared to a standard baseline?
• RQ2 How does skill transfer affect training dynamics, particularly in early training stages?
• RQ3 To what extent does skill transfer improve performance when only a small fraction of the training data is available (e.g., 2%, 5%)?
• RQ4 Does token-wise label supervision for text classification tasks lead to better transfer performance than sentence-wise supervision?
• RQ5 Which individual skills (NER, TE, PPDB, QC) contribute most significantly to improved reading comprehension?

Key findings

• The model with all four skill tasks fine-tuned achieved an F1 score of 60.92 and EM of 48.70 on the SQuAD dev set, outperforming the baseline model without skills (F1: 59.41, EM: 46.90).
• Even with only 2% of the training data, the skill transfer model reached better performance than the baseline or random-weighted models within 1,000 training steps.
• Token-wise label supervision for question type classification significantly outperformed sentence-wise supervision, especially in early training, as shown in Figure 4.
• Ablation studies revealed that removing any single skill led to a drop in performance, with the largest drop observed when removing NER (EM dropped from 48.70 to 48.52), indicating its strong contribution.
• The model with all skills and fine-tuned weights achieved the highest performance, demonstrating that joint skill transfer is effective and synergistic.
• The method showed consistent gains in early training, suggesting that skill transfer accelerates learning and improves sample efficiency.