[Paper Review] Hierarchical Curriculum Learning for AMR Parsing

Peiyi Wang, Liang Chen+5 more|arXiv (Cornell University)|Oct 15, 2021

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes Hierarchical Curriculum Learning (HCL) for AMR parsing, which introduces Structure-level Curriculum (SC) and Instance-level Curriculum (IC) to help sequence-to-sequence models better learn the hierarchical structure of AMR graphs. By training first on core semantic elements and simpler instances before progressing to complex structures, HCL improves model performance, achieving 84.3 smatch on AMR2.0 and 83.7 on AMR3.0, outperforming prior SOTA methods.

ABSTRACT

Abstract Meaning Representation (AMR) parsing aims to translate sentences to semantic representation with a hierarchical structure, and is recently empowered by pretrained sequence-to-sequence models. However, there exists a gap between their flat training objective (i.e., equally treats all output tokens) and the hierarchical AMR structure, which limits the model generalization. To bridge this gap, we propose a Hierarchical Curriculum Learning (HCL) framework with Structure-level (SC) and Instance-level Curricula (IC). SC switches progressively from core to detail AMR semantic elements while IC transits from structure-simple to -complex AMR instances during training. Through these two warming-up processes, HCL reduces the difficulty of learning complex structures, thus the flat model can better adapt to the AMR hierarchy. Extensive experiments on AMR2.0, AMR3.0, structure-complex and out-of-distribution situations verify the effectiveness of HCL.

Motivation & Objective

• To address the gap between flat sequence-to-sequence training objectives and the hierarchical nature of AMR graphs.
• To improve model generalization and performance on complex and out-of-distribution AMR instances.
• To simulate human learning by progressively introducing complexity through structured and instance-based curricula.
• To enhance the ability of pretrained sequence-to-sequence models to capture hierarchical semantic structures in AMR parsing.

Proposed method

• Structure-level Curriculum (SC) trains the model on AMR sub-graphs ordered by increasing depth, starting from core semantic elements near the root.
• Instance-level Curriculum (IC) trains the model on AMR instances ordered by increasing structural complexity, starting with simpler graphs.
• The model is fine-tuned using a curriculum schedule that alternates between SC and IC, gradually increasing the difficulty of training examples.
• Linearized AMR graphs are generated using DFS-based tokenization with special tokens for variables and depth markers.
• The framework is integrated into a BART-based sequence-to-sequence model, with ablation studies to isolate the effect of each curriculum.
• Post-processing follows the same procedure as Bevilacqua et al. (2021) to ensure fair comparison.

Experimental results

Research questions

• RQ1 Can a hierarchical curriculum learning strategy improve the performance of sequence-to-sequence models on AMR parsing?
• RQ2 Does training on core semantic elements before detailed structures enhance model understanding of AMR hierarchy?
• RQ3 Can progressive training on simpler to more complex AMR instances improve generalization on complex and out-of-distribution data?
• RQ4 How do the structure-level and instance-level curricula compare in effectiveness and complementarity?

Key findings

• HCL achieves 84.3 smatch on AMR2.0 and 83.7 on AMR3.0, outperforming the prior SOTA method by 0.5 and 0.7 smatch points, respectively.
• On structure-dependent metrics (unlabeled, re-entrancy, SRL), HCL improves F1 by 2.97 and 2.83 points on AMR2.0 and AMR3.0, respectively, showing enhanced structural learning.
• The model with both curricula outperforms ablated versions, with SC showing a stronger individual effect than IC, indicating SC's greater impact on hierarchical perception.
• On out-of-distribution datasets (BIO, TLP, News3), HCL achieves 61.1, 78.2, and 75.3 smatch, surpassing Bevilacqua et al. (2021) on all three, demonstrating improved generalization.
• Performance gains are most pronounced on deeper AMR graphs, with HCL consistently outperforming the baseline as graph depth increases.