[Paper Review] Mordecai 3: A Neural Geoparser and Event Geocoder

Andrew Halterman|arXiv (Cornell University)|Mar 23, 2023

Geographic Information Systems Studies5 citations

TL;DR

Mordecai 3 is a neural geoparser and event geocoder that uses a transformer-based neural ranking model to resolve place names to Geonames entries and an off-the-shelf question-answering model to identify event locations. It achieves 94.2% country-level accuracy and 82.8% exact match accuracy on benchmark datasets, outperforming prior systems in key metrics while supporting abstention from incorrect geolocation when the correct location is missing.

ABSTRACT

Mordecai3 is a new end-to-end text geoparser and event geolocation system. The system performs toponym resolution using a new neural ranking model to resolve a place name extracted from a document to its entry in the Geonames gazetteer. It also performs event geocoding, the process of linking events reported in text with the place names where they are reported to occur, using an off-the-shelf question-answering model. The toponym resolution model is trained on a diverse set of existing training data, along with several thousand newly annotated examples. The paper describes the model, its training process, and performance comparisons with existing geoparsers. The system is available as an open source Python library, Mordecai 3, and replaces an earlier geoparser, Mordecai v2, one of the most widely used text geoparsers (Halterman 2017).

Motivation & Objective

• To develop a robust, end-to-end geoparsing system that improves accuracy in resolving ambiguous place names in text.
• To address the challenge of event geocoding by linking reported events to their correct geographic locations using contextual reasoning.
• To enhance geoparsing performance through a neural similarity model trained on diverse, newly annotated data and existing datasets.
• To improve robustness in real-world scenarios by enabling the model to detect when the correct location is absent from candidate results.
• To provide an open-source, production-ready Python library replacing Mordecai v2 for broader adoption in social science research.

Proposed method

• Uses spaCy’s transformer-based NER to extract toponyms from text documents.
• Queries a custom Elasticsearch-indexed Geonames gazetteer using fuzzy matching on primary and alternative place names.
• Employs a neural ranking model that computes multiple similarity features, including contextual embeddings and string similarity, to select the best candidate location.
• Leverages off-the-shelf question-answering models to identify the location in a document where an event is reported to occur.
• Trains the neural model on a combination of existing datasets (e.g., GeoWebNews, TR News, LGL) and 2,615 newly annotated examples, including synthetic and Wikipedia-derived data.
• Incorporates features such as population, administrative hierarchy, and alternative name length to improve disambiguation.

Experimental results

Research questions

• RQ1 Can a neural ranking model trained on diverse, high-quality data improve toponym resolution accuracy beyond traditional heuristics?
• RQ2 How well can a pre-trained question-answering model be adapted to identify event locations in text when multiple place names are present?
• RQ3 To what extent can the model detect when the correct location is missing from candidate results, reducing false geolocation errors?
• RQ4 Does combining contextual embeddings with gazetteer metadata improve performance on ambiguous or low-frequency place names?
• RQ5 How does the system perform across diverse text types, including local news and synthetic templates?

Key findings

• Mordecai 3 achieves 94.2% country-level accuracy and 82.8% exact match accuracy across multiple datasets, outperforming prior geoparsers on key metrics.
• On the GWN corpus, it records a mean error of 184 km and an accuracy@161 km of 94%, placing it among the top-performing systems.
• The model correctly identifies unanswerable cases (i.e., missing correct locations) with 70.3% accuracy on training data and up to 100% on newly collected data.
• Performance is strongest on synthetic data (97.4% correct abstention) and weakest on the LGL corpus (40.0% correct abstention), reflecting challenges in local US news with ambiguous names.
• The system maintains a median error of 0 km across all datasets, indicating that while most predictions are correct, some high-impact errors (very large distances) still occur.
• The model shows strong performance on the TR and GWN datasets, with exact match accuracies of 82.8% and 91.2%, respectively, and high country-level accuracy (95.8% on GWN).