[Paper Review] Semantic Frame Parsing for Information Extraction : the CALOR corpus

Gabriel Marzinotto, Jérémy Auguste+3 more|arXiv (Cornell University)|Dec 19, 2018

Natural Language Processing Techniques10 references4 citations

TL;DR

This paper introduces the CALOR corpus, a 1.3M-word French-annotated dataset for partial semantic frame parsing tailored to information extraction. It compares CRF and biLSTM multi-task models for joint frame identification and semantic role labeling, showing that biLSTM-MT achieves higher recall while CRF-MM offers better precision due to reduced label ambiguity in subtasks.

ABSTRACT

This paper presents a publicly available corpus of French encyclopedic history texts annotated according to the Berkeley FrameNet formalism. The main difference in our approach compared to previous works on semantic parsing with FrameNet is that we are not interested here in full text parsing but rather on partial parsing. The goal is to select from the FrameNet resources the minimal set of frames that are going to be useful for the applicative framework targeted, in our case Information Extraction from encyclopedic documents. Such an approach leverages the manual annotation of larger corpora than those obtained through full text parsing and therefore opens the door to alternative methods for Frame parsing than those used so far on the FrameNet 1.5 benchmark corpus. The approaches compared in this study rely on an integrated sequence labeling model which jointly optimizes frame identification and semantic role segmentation and identification. The models compared are CRFs and multitasks bi-LSTMs.

Motivation & Objective

• To develop a cost-effective, large-scale corpus for semantic frame parsing focused on information extraction rather than full text annotation.
• To address the limitations of FrameNet 1.5, which suffers from sparse frame distribution and high annotation cost in full parsing.
• To evaluate sequence labeling models—CRF and biLSTM—on a partial parsing setup with a limited, domain-relevant frame inventory.
• To compare the performance of integrated models that jointly predict frame triggers and semantic roles versus multi-step approaches.
• To investigate how model architecture affects precision-recall trade-offs in frame parsing under partial annotation.

Proposed method

• The CALOR corpus is constructed from four diverse French sources: Wikipedia portals (archaeology, WW1), Vikidia (children’s encyclopedia), and ClioTexte (WW1 historical documents), totaling 1.3M words.
• Only a minimal, application-specific subset of FrameNet frames and lexical units is annotated, focusing on relevance to information extraction in historical texts.
• A joint sequence labeling framework is used, where each token is assigned a label combining frame trigger and semantic role, with a heuristic filter to maintain structural coherence.
• Two models are evaluated: a multi-model CRF (CRF-MM) trained per lexical unit to reduce label ambiguity, and a multi-task biLSTM (biLSTM-MT) sharing features across all LUs.
• The models are trained and evaluated using precision, recall, and F-measure at the Equal Error Rate (EER) operating point across four subtasks: trigger identification, trigger classification, role identification, and role classification.
• Feature sets include word forms, POS tags, and contextual embeddings, with model performance assessed via precision-recall curves and F-measure at EER.

Experimental results

Research questions

• RQ1 Can partial frame parsing with a limited frame inventory enable the creation of larger, more cost-effective annotated corpora than full parsing?
• RQ2 How do CRF and biLSTM models compare in performance when jointly predicting frame triggers and semantic roles in a partial parsing setup?
• RQ3 Does the multi-task biLSTM model outperform the multi-model CRF in recall and generalization due to shared feature learning across frames?
• RQ4 To what extent does model architecture influence the precision-recall trade-off in frame parsing tasks?
• RQ5 Can models trained on one domain (e.g., archaeology) generalize to another (e.g., WW1) with different writing styles?

Key findings

• The biLSTM-MT model achieves higher F-measure on role identification (70.3) and role classification (63.2) compared to CRF-MM (69.7 and 60.6), indicating better recall and robustness in complex syntactic patterns.
• The CRF-MM model outperforms biLSTM-MT in trigger identification (96.4 vs. 95.5 F1) and trigger classification (95.2 vs. 92.3 F1), due to reduced label ambiguity from per-LU training.
• The biLSTM-MT model exhibits a better recall-precision balance, capturing more frame elements despite higher precision errors, due to shared feature learning across all LUs.
• The CRF-MM model achieves higher precision but lower recall, indicating a more conservative prediction strategy with fewer false positives.
• The CALOR corpus enables large-scale frame annotation at lower cost than full parsing, with significantly more examples per frame than in FrameNet 1.5, supporting better model training.
• The results confirm that partial parsing with a domain-specific frame set is viable and effective for information extraction, especially when using deep learning models that benefit from shared representations.