[논문 리뷰] BERT-of-Theseus: Compressing BERT by Progressive Module Replacing

Canwen Xu, Wangchunshu Zhou외 3명|arXiv (Cornell University)|2020. 02. 07.

Topic Modeling참고 문헌 44인용 수 34

한 줄 요약

BERT-of-Theseus는 학습 중에 BERT 모듈을 점진적으로 더 작은 대체 모듈로 교체하여 GLUE에서 BERT-base 성능의 98% 이상을 유지하면서 약 1.94배 속도 향상을 달성하고, 추가 증류 손실 없이 수행한다.

ABSTRACT

In this paper, we propose a novel model compression approach to effectively compress BERT by progressive module replacing. Our approach first divides the original BERT into several modules and builds their compact substitutes. Then, we randomly replace the original modules with their substitutes to train the compact modules to mimic the behavior of the original modules. We progressively increase the probability of replacement through the training. In this way, our approach brings a deeper level of interaction between the original and compact models. Compared to the previous knowledge distillation approaches for BERT compression, our approach does not introduce any additional loss function. Our approach outperforms existing knowledge distillation approaches on GLUE benchmark, showing a new perspective of model compression.

연구 동기 및 목표

• 대형 Transformer 모델인 BERT와 같은 모델의 크기와 계산 요구를 추가 증류 손실에 의존하지 않고 줄이는 동기를 제시한다.
• 학습 중에 선행 모듈과 후속 모듈을 교차하는 점진적 모듈 교체(Theseus Compression) 프레임워크를 도입한다.
• 결과적으로 압축된 모델이 GLUE에서 원래의 성능에 근접하면서 속도 향상을 제공하는지 입증한다.
• 커리큘럼 기반 교체 스케줄링이 일정한 교체 비율보다 더 나은 결과를 낳는지 보여준다.
• 어떤 층과 교체 전략이 성능에 가장 큰 영향을 미치는지 분석한다.]
• method [...]
• research_questions:['Can progressively replacing modules within a large pretrained model yield effective compression without extra distillation objectives?','Does curriculum-based scheduling improve the trade-off between compression and accuracy compared to constant replacement rates?','How does Theseus Compression compare to KD-based baselines on GLUE in terms of performance, speed, and model size?','Is the approach model-agnostic to Transformer-based architectures and potential to apply to other domains?']
• key_findings':['BERT-of-Theseus는 6-레이어 압축 모델에서 1.94x 추론 속도 향상을 달성하면서 GLUE에서 BERT-base 성능의 98.4%(개발 세트)와 98.3%(테스트 세트)를 유지한다.','본 방법은 GLUE의 대부분 작업에서 vanilla KD 및 PKD 기준선보다 성능이 더 우수하다.','커리큘럼 교체 스케줄러는 일정한 교체 및 반 커리큘럼 전략보다 지속적으로 성능을 향상시킨다.','초기 Transformer 층을 교체하는 것이 후기 층 교체보다 성능 저하를 더 많이 유발하는 경향이 있어 초기 층이 언어 특징에 더 기여함을 시사한다.','중간 작업 전이(MNLI를 사전학습 작업으로 사용)가 DistilBERT 및 PD-BERT에 비해 여러 작업에서 경쟁력 있거나 우수한 결과를 보여준다.']
• table_headers':['방법','CoLA','MNLI','MRPC','QNLI','QQP','RTE','SST-2','STS-B','매크로'],
• table_rows1":[["BERT-base","54.3","83.5","89.5","91.2","89.8","71.1","91.5","88.9","82.5"]]
• research_questions: ["Can progressively replacing modules within a large pretrained model yield effective compression without extra distillation objectives?", "Does curriculum-based scheduling improve the trade-off between compression and accuracy compared to constant replacement rates?", "How does Theseus Compression compare to KD-based baselines on GLUE in terms of performance, speed, and model size?", "Is the approach model-agnostic to Transformer-based architectures and potential to apply to other domains?"]
• key_findings: ["BERT-of-Theseus achieves 1.94x inference speed-up with 6-layer compressed models while retaining 98.4% (development) and 98.3% (test) of BERT-base performance on GLUE.", " The method outperforms vanilla KD and PKD baselines on GLUE across most tasks.", " A curriculum replacement scheduler consistently improves performance over constant replacement and anti-curriculum strategies.", " Replacing earlier Transformer layers tends to hurt performance more than replacing later layers, indicating early layers contribute more to linguistic features.", " Intermediate-task transfer (MNLI as a pretraining task) shows competitive or superior results on several tasks compared to DistilBERT and PD-BERT."]
• table_headers: ["Method","CoLA","MNLI","MRPC","QNLI","QQP","RTE","SST-2","STS-B","Macro"]
• table_rows: [["BERT-base","54.3","83.5","89.5","91.2","89.8","71.1","91.5","88.9","82.5"], ["DistilBERT","43.6","79.0","87.5","85.3","84.9","59.9","90.7","81.2","76.5"], ["Vanilla KD","45.1","80.1","86.2","88.0","88.1","64.9","90.5","84.9","78.5"], ["BERT-PKD","45.5","81.3","85.7","88.4","88.4","66.5","91.3","86.2","79.2"], ["BERT-of-Theseus","51.1","82.3","89.0","89.5","89.6","68.2","91.5","88.7","81.2"]]} }<|vq_13295|>}}}6|} ۵} 6]} 6} }} 6]} } } } } } } }}}} } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } } }

제안 방법

• Partition the original BERT into modules and define compact substitute modules for each.
• During training, replace predecessor modules with successor modules with a probability p, mixing both in a single forward pass.
• Optimize only the task-specific loss (e.g., cross-entropy) while freezing predecessor embeddings and outputs to enable gradient flow across modules.
• After convergence, assemble all successor modules into a full successor model and fine-tune it with the same loss.
• Apply a Curriculum Learning-driven replacement scheduler to progressively increase the replacement probability over training steps.
• Provide a simple linear scheduler p_d = min(1, kt + b) to control the dynamic replacement rate and warm up learning.

실험 결과

연구 질문

• RQ1 Can progressively replacing modules within a large pretrained model yield effective compression without extra distillation objectives?
• RQ2 Does curriculum-based scheduling improve the trade-off between compression and accuracy compared to constant replacement rates?
• RQ3 How does Theseus Compression compare to KD-based baselines on GLUE in terms of performance, speed, and model size?
• RQ4 Is the approach model-agnostic to Transformer-based architectures and potential to apply to other domains?

주요 결과

• BERT-of-Theseus achieves 1.94x inference speed-up with 6-layer compressed models while retaining 98.4% (development) and 98.3% (test) of BERT-base performance on GLUE.
• The method outperforms vanilla KD and PKD baselines on GLUE across most tasks.
• A curriculum replacement scheduler consistently improves performance over constant replacement and anti-curriculum strategies.
• Replacing earlier Transformer layers tends to hurt performance more than replacing later layers, indicating early layers contribute more to linguistic features.
• Intermediate-task transfer (MNLI as a pretraining task) shows competitive or superior results on several tasks compared to DistilBERT and PD-BERT.