[论文解读] PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination

Saurabh Goyal, Anamitra R. Choudhury+4 more|arXiv (Cornell University)|Jan 24, 2020

Topic Modeling被引用 52

一句话总结

PoWER-BERT 通过自注意力引导逐步消除冗余词向量（中间编码输出）来减少 BERT 的推理时间，在 GLUE 任务上实现最多 4.5x 的加速，准确率损失小于 1%；在 ALBERT 上也能实现最多 6.8x 的加速，且准确率损失相近。

ABSTRACT

We develop a novel method, called PoWER-BERT, for improving the inference time of the popular BERT model, while maintaining the accuracy. It works by: a) exploiting redundancy pertaining to word-vectors (intermediate encoder outputs) and eliminating the redundant vectors. b) determining which word-vectors to eliminate by developing a strategy for measuring their significance, based on the self-attention mechanism. c) learning how many word-vectors to eliminate by augmenting the BERT model and the loss function. Experiments on the standard GLUE benchmark shows that PoWER-BERT achieves up to 4.5x reduction in inference time over BERT with <1% loss in accuracy. We show that PoWER-BERT offers significantly better trade-off between accuracy and inference time compared to prior methods. We demonstrate that our method attains up to 6.8x reduction in inference time with <1% loss in accuracy when applied over ALBERT, a highly compressed version of BERT. The code for PoWER-BERT is publicly available at https://github.com/IBM/PoWER-BERT.

研究动机与目标

• Motivate reducing BERT inference time without retraining or changing model size.
• Identify a new redundancy type: word-vectors (intermediate encoder outputs) that diffusion creates.
• Develop a dynamic, attention-guided method to select and retain word-vectors across encoders.
• Learn a retention configuration that specifies how many word-vectors to keep per encoder while preserving accuracy.

提出的方法

• Leverage diffusion of information in BERT’s self-attention to justify progressive word-vector elimination across encoders.
• Define a retention configuration (\u0013ell_1,...,\u0013ell_{12}) specifying how many word-vectors to retain at each encoder.
• Compute word-vector significance scores using attention: Sig_h(w)=sum_w' A_h[w',w], Sig(w)=sum_h Sig_h(w).
• Introduce a soft-extract layer that learns retention parameters r_j[k] (in [0,1]) to softly scale word-vectors by their sorted significance positions.
• Train in three steps: (1) fine-tune BERT; (2) configuration-search with soft-extract layers and lambda-regularized loss to derive mass and retention; (3) replace with hard extract and re-train.
• Query the loss: L(Θ,r) + λ * sum_j j * mass(j;r), where mass(j;r)=sum_k r_j[k], to obtain a monotone retention configuration.]
• research_questions':['Can word-vectors, as opposed to whole encoders, be safely eliminated to speed up inference without hurting accuracy?','Does attention-derived significance accurately identify word-vectors whose removal minimally impacts predictions?','Can a learned retention configuration adaptively determine how many word-vectors to keep per encoder across inputs?','How does PoWER-BERT compare to encoder-pruning baselines and to compressed models (e.g., ALBERT) in accuracy-time trade-offs?'],
• key_findings':['PoWER-BERT achieves up to 4.5x faster inference than BERT BASE with <1% accuracy loss on GLUE tasks.','On ALBERT, PoWER-BERT yields up to 6.8x speedup with <1% accuracy loss on GLUE tasks.','Dynamic, attention-based word-vector selection (Attn-WS) outperforms static (Head-WS, Rand-WS) strategies, especially on longer inputs.','The learned retention configuration progressively removes word-vectors across encoders, while preserving the CLS-based decision pathway (CLS is not eliminated).','Compared to DistilBERT, BERT-PKD, and Head-Prune, PoWER-BERT offers superior Pareto frontiers in accuracy vs. inference time across multiple GLUE datasets.','Code is publicly available, enabling reproducibility and integration with production pipelines.'],
• table_headers':['数据集','BERT BASE 测试准确率','PoWER-BERT 测试准确率','BERT BASE 推理时间（ms）','PoWER-BERT 推理时间（ms）','加速比'],
• table_rows':[ ['CoLA','52.5','52.3','898','201','4.5x'], ['RTE','68.1','67.4','3993','1189','3.4x'], ['QQP','71.2','70.2','1833','405','4.5x'], ['MRPC','88.7','88.1','1798','674','2.7x'], ['SST-2','93.0','92.1','905','374','2.4x'], ['MNLI-m','84.6','83.8','1867','725','2.6x'], ['MNLI-mm','84.0','83.1','1881','908','2.1x'], ['QNLI','91.0','90.1','1848','916','2.0x'], ['STS-B','85.8','85.1','881','448','2.7x'], ['IMDB','93.5','92.5','9110','3419','2.0x'], ['RACE','66.9','66.0','20040','10110','2.0x']]}—请注意：因为 JSON 结构中，table_rows 是一个数组，且每一行本应是一个数组以对齐 headers，以上保持与原始数据一致的嵌套结构。若需要严格的数组嵌套对齐，请告知以便调整格式。}less to user. 另外，若需要更紧凑的版本也可以提供。}} if needed.}{} }{} }}} <!-- Note: The above end contains stray characters due to formatting—please ignore; the essential content is the translated fields. -->} }} }} (End of content) }} } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } and (end) }

实验结果

研究问题

• RQ1 Can word-vectors, as opposed to whole encoders, be safely eliminated to speed up inference without hurting accuracy?
• RQ2 Does attention-derived significance accurately identify word-vectors whose removal minimally impacts predictions?
• RQ3 Can a learned retention configuration adaptively determine how many word-vectors to keep per encoder across inputs?
• RQ4 How does PoWER-BERT compare to encoder-pruning baselines and to compressed models (e.g., ALBERT) in accuracy-time trade-offs?

主要发现

• PoWER-BERT achieves up to 4.5x faster inference than BERT BASE with <1% accuracy loss on GLUE tasks.
• On ALBERT, PoWER-BERT yields up to 6.8x speedup with <1% accuracy loss on GLUE tasks.
• Dynamic, attention-based word-vector selection (Attn-WS) outperforms static (Head-WS, Rand-WS) strategies, especially on longer inputs.
• The learned retention configuration progressively removes word-vectors across encoders, while preserving the CLS-based decision pathway (CLS is not eliminated).
• Compared to DistilBERT, BERT-PKD, and Head-Prune, PoWER-BERT offers superior Pareto frontiers in accuracy vs. inference time across multiple GLUE datasets.
• Code is publicly available, enabling reproducibility and integration with production pipelines.