[Paper Review] Introducing the ICBe Dataset: Very High Recall and Precision Event Extraction from Narratives about International Crises

Rex W. Douglass, Thomas Leo Scherer+11 more|arXiv (Cornell University)|Feb 14, 2022

Computational and Text Analysis Methods4 citations

TL;DR

This paper introduces ICBe, a high-precision, high-recall event extraction dataset for international crises using a novel ontology applied to high-quality narratives from the International Crisis Behavior (ICB) project. By combining human coding with NLP techniques, ICBe achieves superior coverage and accuracy over existing datasets, validated through case studies of the Cuban Missile Crisis and Crimea-Donbas Crisis, and includes new visualizations and benchmarking tools for event extraction evaluation.

ABSTRACT

How do international crises unfold? We conceptualize of international relations as a strategic chess game between adversaries and develop a systematic way to measure pieces, moves, and gambits accurately and consistently over a hundred years of history. We introduce a new ontology and dataset of international events called ICBe based on a very high-quality corpus of narratives from the International Crisis Behavior (ICB) Project. We demonstrate that ICBe has higher coverage, recall, and precision than existing state of the art datasets and conduct two detailed case studies of the Cuban Missile Crisis (1962) and Crimea-Donbas Crisis (2014). We further introduce two new event visualizations (event icongraphy and crisis maps), an automated benchmark for measuring event recall using natural language processing (sythnetic narratives), and an ontology reconstruction task for objectively measuring event precision. We make the data, online appendix, replication material, and visualizations of every historical episode available at a companion website www.crisisevents.org and the github repository.

Motivation & Objective

• To develop a systematic, high-accuracy method for measuring strategic interactions in international relations over 100 years of crisis history.
• To address the limitations of existing event datasets, which suffer from low recall and precision in reconstructing historical crisis narratives.
• To create a new gold-standard dataset (ICBe) with enhanced coverage and consistency through a hybrid human-and-NLP coding process.
• To introduce objective evaluation mechanisms—such as synthetic narratives and ontology reconstruction—for measuring recall and precision in event extraction.
• To provide open-access visualizations, replication materials, and data via www.crisisevents.org and GitHub for reproducibility and reuse.

Proposed method

• The ICBe dataset is constructed by applying a newly designed, flexible ontology to 471 high-quality historical narratives from the ICB project, mapping each sentence to discrete, machine-readable events.
• A hybrid coding process combines expert human coders with natural language processing to ensure consistency and scalability in event extraction.
• The ontology is designed to capture fine-grained diplomatic, military, and political behaviors with structured event types, participants, times, and locations.
• A novel benchmark using synthetic narratives is developed to objectively measure recall by assessing how well event codings reconstruct original narrative content.
• Event precision is evaluated through an ontology reconstruction task, where coders re-encode events based on extracted data to test consistency and accuracy.
• The method includes two new visualizations: event icongraphy and crisis maps, to enhance interpretability and temporal-spatial understanding of crisis dynamics.

Experimental results

Research questions

• RQ1 To what extent can event extraction systems reconstruct the full narrative content of historical international crises with high recall and precision?
• RQ2 How does the ICBe dataset compare to existing state-of-the-art event datasets in terms of coverage, recall, and precision?
• RQ3 Can synthetic narratives serve as a valid and objective benchmark for measuring event extraction recall in international relations data?
• RQ4 To what extent does the ICBe ontology capture the strategic complexity of international crises compared to prior event coding frameworks?
• RQ5 How do visualizations like event icongraphy and crisis maps improve the interpretability of large-scale event data in crisis analysis?

Key findings

• ICBe achieves significantly higher recall and precision than existing state-of-the-art event datasets, demonstrating superior coverage of historical crisis narratives.
• The synthetic narrative benchmark reveals that current systems fail to reconstruct the fine-grained variation in international crises, indicating substantial under-recognition of key events.
• The ontology reconstruction task confirms that ICBe codings are highly consistent and precise, with minimal deviation from original narrative content.
• Case studies of the Cuban Missile Crisis (1962) and Crimea-Donbas Crisis (2014) show that ICBe captures critical diplomatic and military moves that are often missed in other datasets.
• The event icongraphy and crisis maps visualizations effectively reveal temporal and spatial patterns in crisis dynamics, enabling deeper analytical insights.
• All data, code, visualizations, and replication materials are publicly available at www.crisisevents.org and on GitHub, ensuring full reproducibility and reuse.