[論文レビュー] LasUIE: Unifying Information Extraction with Latent Adaptive Structure-aware Generative Language Model

Hao Fei, Shengqiong Wu+7 more|arXiv (Cornell University)|Apr 13, 2023

Topic Modeling被引用数 69

ひとこと要約

LasUIE は、単一の生成型言語モデルのもとで情報抽出タスクを統合し、無監督の構造誘導子と構造ブロードキャスターを通じて潜在的で構造認識的な表現を学習し、タスク指向のファインチューニングを行うことにより、12 の IE ベンチマーク全体で一貫した改善を達成する。

ABSTRACT

Universally modeling all typical information extraction tasks (UIE) with one generative language model (GLM) has revealed great potential by the latest study, where various IE predictions are unified into a linearized hierarchical expression under a GLM. Syntactic structure information, a type of effective feature which has been extensively utilized in IE community, should also be beneficial to UIE. In this work, we propose a novel structure-aware GLM, fully unleashing the power of syntactic knowledge for UIE. A heterogeneous structure inductor is explored to unsupervisedly induce rich heterogeneous structural representations by post-training an existing GLM. In particular, a structural broadcaster is devised to compact various latent trees into explicit high-order forests, helping to guide a better generation during decoding. We finally introduce a task-oriented structure fine-tuning mechanism, further adjusting the learned structures to most coincide with the end-task's need. Over 12 IE benchmarks across 7 tasks our system shows significant improvements over the baseline UIE system. Further in-depth analyses show that our GLM learns rich task-adaptive structural bias that greatly resolves the UIE crux, the long-range dependence issue and boundary identifying. Source codes are open at https://github.com/ChocoWu/LasUIE.

研究の動機と目的

• universal information extraction (IE) modeling via a single generative language model (GLM).
• syntactic information by learning latent heterogeneous structures to aid IE.
• Develop a three-stage training pipeline including unsupervised structure-aware post-training and task-oriented fine-tuning.
• Demonstrate improvements across multiple IE tasks and resource settings using a unified UIE framework.

提案手法

• Transform all IE tasks into a unified text-generation problem by predicting a linearized hierarchical expression of spans, relations, and their attributes.
• Introduce a heterogeneous structure inductor (HSI) to unsupervisedly induce constituency and dependency structures on top of a GLM encoder.
• Implement a structural broadcaster (SB) that compacts multiple latent trees into explicit constituency-like and dependency-like forests to guide decoding.
• Apply a task-oriented structure fine-tuning stage using policy gradient to adapt induced structures to end-task needs.
• Base the model on a Transformer encoder-decoder GLM (e.g., T5), with a three-stage training process: pre-training, unsupervised structure-aware post-training, and supervised task fine-tuning.
• Utilize losses L_W (language modeling), L_D (dependency syntax), L_C (constituency syntax), and L_SDR (structure diversification) during post-training, plus L_Task and task-specific structural adjustments during fine-tuning.

実験結果

リサーチクエスチョン

• RQ1 latent, jointly learned constituency and dependency structures improve UIE beyond using external syntax trees?
• RQ2 How do two heterogeneous structures complement each other for boundary detection and long-range dependency in IE?
• RQ3 Is automatically induced latent structure more effective than injecting external parse trees for UIE?
• RQ4 Does task-oriented structural fine-tuning further boost UIE performance across tasks and data regimes?

主な発見

• LasUIE consistently outperforms baseline UIE and other SOTA models across 12 IE benchmarks, under both fully supervised and low-resource settings.
• Integrating two heterogeneous structures (constituency and dependency) yields complementary gains, with constituency aiding boundary detection and dependency aiding long-range dependencies.
• Latent structure learning with SB and SDR outperforms external syntax injection, showing robustness to noisy parsed trees.
• Structural fine-tuning via policy gradients further improves end-task performance, with ablations showing significant drops without SB or SDR.
• Low-resource transfer benefits notably from structure-informed GLMs, especially when using unified IE modeling.