[論文レビュー] Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences

Yikuan Li, Ramsey M. Wehbe+3 more|arXiv (Cornell University)|Jan 27, 2022

Topic Modeling被引用数 68

ひとこと要約

本論文は、Clinical-LongformerとClinical-BigBirdの2つの長系列トランスフォーマーを大規模な臨床ノート上で事前学習させ、ClinicalBERTおよび他の短序列モデルを、複数の臨床NLPタスクで上回ることを示します。

ABSTRACT

Transformers-based models, such as BERT, have dramatically improved the performance for various natural language processing tasks. The clinical knowledge enriched model, namely ClinicalBERT, also achieved state-of-the-art results when performed on clinical named entity recognition and natural language inference tasks. One of the core limitations of these transformers is the substantial memory consumption due to their full self-attention mechanism. To overcome this, long sequence transformer models, e.g. Longformer and BigBird, were proposed with the idea of sparse attention mechanism to reduce the memory usage from quadratic to the sequence length to a linear scale. These models extended the maximum input sequence length from 512 to 4096, which enhanced the ability of modeling long-term dependency and consequently achieved optimal results in a variety of tasks. Inspired by the success of these long sequence transformer models, we introduce two domain enriched language models, namely Clinical-Longformer and Clinical-BigBird, which are pre-trained from large-scale clinical corpora. We evaluate both pre-trained models using 10 baseline tasks including named entity recognition, question answering, and document classification tasks. The results demonstrate that Clinical-Longformer and Clinical-BigBird consistently and significantly outperform ClinicalBERT as well as other short-sequence transformers in all downstream tasks. We have made our source code available at [https://github.com/luoyuanlab/Clinical-Longformer] the pre-trained models available for public download at: [https://huggingface.co/yikuan8/Clinical-Longformer].

研究の動機と目的

• 退院要約などの長文臨床テキストの存在を踏まえ、臨床NLPにおける長系列トランスフォーマーの必要性を動機づける。
• 約2百万件のMIMIC-III臨床ノート上で、ドメイン強化された長系列モデルを事前学習する。
• QA、NER、文書分類など、多様な臨床NLPタスクでモデルを評価する。
• 再現性と再利用のために、公開コードと事前学習済みウェイトを提供する。

提案手法

• LongformerとBigBirdの派生モデルを、ベースのLongformerおよびITC BigBirdの重みから、MIMIC-IIIの約2M臨床ノートを対象にバイトレベルBPEを用いて事前学習する。
• FP16で6×32GBのGPUを使用; Clinical-Longformerを200kステップ、Clinical-BigBirdを300kステップで学習; 学習率3e-5。
• QA、NER、分類タスクを含む9つの臨床NLPデータセットでファインチューニングを行い、最大入力長4096トークン、必要に応じてスニペットプーリングで集約する。
• 各タスクでBERT、BioBERT、ClinicalBERTをベースラインと比較する。

実験結果

リサーチクエスチョン

• RQ1 長いシーケンスのトランスフォーマーは、512トークンを超える長さの入力を活用して、臨床テキストタスクの性能を向上させることができるか？
• RQ2 Clinical-LongformerとClinical-BigBirdは、臨床領域のQA、NER、文書分類の各タスクで、短序列のドメインモデルを上回るか？
• RQ3 臨床NLPベンチマークにおいて、二つの長序列モデルはパフォーマンスと学習効率の点でどのように比較されるか？

主な発見

事前学習済みモデル	emrQA-Medication EM	emrQA-Medication F1	emrQA-Relation EM	emrQA-Relation F1	emrQA-HeartDisease EM	emrQA-HeartDisease F1
BERT	0.240	0.675	0.833	0.924	0.650	0.698
BioBERT	0.247	0.700	0.836	0.926	0.647	0.702
ClinicalBERT	0.297	0.698	0.849	0.929	0.666	0.711
Clinical-Longformer	0.302	0.716	0.911	0.948	0.698	0.734
Clinical-Big Bird	0.300	0.715	0.898	0.944	0.664	0.711

• Both Clinical-Longformer and Clinical-BigBird outperform short-sequence baselines (ClinicalBERT, BioBERT, BERT) across the evaluated tasks.
• They achieve notable gains in QA (F1 up to 0.948 on Relation), NER (F1 up to 0.887 on i2b2 2010), and document classification on several clinical datasets.
• Clinical-Longformer generally yields better results than Clinical-BigBird and is recommended when resources are limited due to faster fine-tuning.
• Long-sequence models show larger gains on datasets with longer average sequence lengths (e.g., i2b2 2014, emrQA HeartDisease).
• Open-source release of code and downloadable pre-trained weights facilitates replication and application to other clinical NLP tasks.