[Paper Review] Delaying Interaction Layers in Transformer-based Encoders for Efficient Open Domain Question Answering

Wissam Siblini, Mohamed Challal+1 more|arXiv (Cornell University)|Oct 16, 2020

Topic Modeling41 references4 citations

TL;DR

This paper proposes Delaying Interaction Layers (DIL) in BERT and ALBERT models to improve efficiency in open-domain question answering (ODQA) by deferring self-attention computations to later layers, reducing FLOPs by up to 85% and achieving up to 10x speedup on CPU/GPU while maintaining or improving performance on multi-passage ODQA benchmarks like OpenSQuAD.

ABSTRACT

Open Domain Question Answering (ODQA) on a large-scale corpus of documents (e.g. Wikipedia) is a key challenge in computer science. Although transformer-based language models such as Bert have shown on SQuAD the ability to surpass humans for extracting answers in small passages of text, they suffer from their high complexity when faced to a much larger search space. The most common way to tackle this problem is to add a preliminary Information Retrieval step to heavily filter the corpus and only keep the relevant passages. In this paper, we propose a more direct and complementary solution which consists in applying a generic change in the architecture of transformer-based models to delay the attention between subparts of the input and allow a more efficient management of computations. The resulting variants are competitive with the original models on the extractive task and allow, on the ODQA setting, a significant speedup and even a performance improvement in many cases.

Motivation & Objective

• Address the high computational cost of transformer-based models in open-domain question answering (ODQA), where processing millions of document passages is infeasible in real time.
• Overcome the bottleneck of slow inference in extractive QA models when applied to large-scale corpora like Wikipedia.
• Explore architectural modifications that reduce computation without sacrificing answer accuracy, especially in multi-passage settings.
• Develop a generic, pre-trained model-compatible method that enables faster inference while preserving performance on standard benchmarks.
• Investigate whether delaying attention interactions improves generalization to irrelevant or multiple passages, reducing overfitting to single relevant documents.

Proposed method

• Introduce a Delayed Interaction Layers (DIL) mechanism that restricts self-attention computation to only the final k layers of the transformer encoder, while the first (12-k) layers process input without cross-segment attention.
• Apply the DIL mechanism to both BERT and ALBERT architectures, creating DilBERT and DilALBERT variants that retain pre-trained weights and are fine-tuned only on downstream QA tasks.
• Use a standard retriever (Lucene with BM25) to retrieve top-p relevant documents, then apply the DilBERT/DilALBERT reader to extract answers from these passages.
• Limit attention to only the last k layers (e.g., k=2) to drastically reduce FLOPs and memory usage during inference, especially beneficial in CPU or low-resource settings.
• Maintain the same model architecture and head structure as the original models, ensuring compatibility with existing pre-trained weights and fine-tuning procedures.
• Train the DIL models on SQuAD v1.1 and fine-tune on OpenSQuAD, using standard cross-entropy loss for span prediction, while monitoring performance on both single- and multi-passage settings.

Experimental results

Research questions

• RQ1 Can deferring self-attention computations to later layers in transformer encoders significantly reduce inference time in open-domain QA without degrading answer accuracy?
• RQ2 Does the DIL mechanism improve generalization to multiple or irrelevant passages, reducing overfitting to single relevant documents?
• RQ3 To what extent does the DIL mechanism preserve performance on standard extractive QA benchmarks like SQuAD v1.1 compared to the original models?
• RQ4 How does the performance of DIL variants compare to standard BERT/ALBERT in the multi-passage ODQA setting, particularly on OpenSQuAD?
• RQ5 Can the DIL mechanism be applied generically to other transformer-based models without requiring retraining or architectural redesign?

Key findings

• The DilBERT and DilALBERT models achieve up to a 10x speedup in inference time on both CPU and GPU when processing ODQA tasks, with a 6x (85%) reduction in FLOPs when k=10.
• On the SQuAD v1.1 benchmark, DilBERT and DilALBERT show competitive performance, with only a slight drop in exact match (EM) and F1 on single-passage QA, indicating minimal degradation on standard tasks.
• On the OpenSQuAD benchmark, DilBERT and DilALBERT outperform their base counterparts when selecting answers from multiple relevant passages, demonstrating improved generalization.
• The DIL mechanism reduces overfitting to single relevant passages, as evidenced by more consistent predictions across multiple passages and reduced sensitivity to irrelevant context.
• The authors report improved results on OpenSQuAD for the Lucene + BERT base baseline due to engineering optimizations, but the DIL variants still achieve state-of-the-art performance with significantly faster inference.
• The method is generic and compatible with existing pre-trained models, enabling deployment without retraining or access to large-scale pre-training data.