[Paper Review] A Study of Geolocation Databases

Yuval Shavitt, Noa Zilberman|arXiv (Cornell University)|May 31, 2010

Geographic Information Systems Studies25 references20 citations

TL;DR

This paper evaluates the accuracy of commercial and academic IP geolocation databases using a novel method that groups IP addresses into Points of Presence (PoPs) based on network structure and delay measurements, enabling high-confidence geolocation validation. The study reveals widespread inaccuracies—with up to 40% of PoP-internal IPs mapped to locations outside 100km radius—highlighting that geolocation databases cannot be trusted as ground truth, even when results are highly correlated.

ABSTRACT

The geographical location of Internet IP addresses has an importance both for academic research and commercial applications. Thus, both commercial and academic databases and tools are available for mapping IP addresses to geographic locations. Evaluating the accuracy of these mapping services is complex since obtaining diverse large scale ground truth is very hard. In this work we evaluate mapping services using an algorithm that groups IP addresses to PoPs, based on structure and delay. This way we are able to group close to 100,000 IP addresses world wide into groups that are known to share a geo-location with high confidence. We provide insight into the strength and weaknesses of IP geolocation databases, and discuss their accuracy and encountered anomalies.

Motivation & Objective

• To assess the accuracy of commercial and academic IP geolocation databases, which are widely used but lack reliable ground truth for validation.
• To develop a method for identifying co-located IP addresses with high confidence using network delay and topology, enabling indirect ground truth for geolocation evaluation.
• To investigate discrepancies among multiple databases and identify systemic errors, such as defaulting to ISP headquarters for large IP blocks.
• To evaluate the performance of measurement-based geolocation tools like Spotter and compare them to database-based approaches.
• To provide insights into the reliability of geolocation data for security, research, and application purposes.

Proposed method

• Grouping IP addresses into PoPs using a hybrid algorithm based on network delay and topological structure, minimizing false positives for non-co-located IPs.
• Using the PoP grouping as a proxy for ground truth: if multiple IPs in the same PoP are mapped to different locations, the database is deemed inconsistent.
• Measuring database accuracy by computing the radius within which all IPs in a PoP are mapped—smaller radii indicate higher accuracy.
• Comparing multiple databases on the same PoP sets to detect consensus and identify outliers, with majority voting used to infer likely correct location.
• Evaluating measurement-based geolocation (e.g., Spotter) by analyzing convergence and agreement across regions and radii.
• Analyzing regional performance differences, particularly between Europe and the USA, to assess the impact of vantage point distribution on accuracy.

Experimental results

Research questions

• RQ1 How accurate are commercial and academic IP geolocation databases when validated against a proxy ground truth derived from PoP grouping?
• RQ2 To what extent do geolocation databases agree with each other, and what proportion of IP addresses are incorrectly mapped to locations thousands of kilometers apart?
• RQ3 How does the performance of measurement-based geolocation (e.g., Spotter) compare to database-based approaches, especially across different geographic regions?
• RQ4 Why do some databases consistently assign the same default location (e.g., ISP headquarters) to large blocks of IP addresses, and how does this affect accuracy?
• RQ5 Can PoP-based grouping serve as a reliable method for validating geolocation databases without requiring external ground truth?

Key findings

• Over 40% of IP addresses within the same PoP were mapped to locations outside a 100km radius by Netacuity, indicating significant inaccuracy despite its high cost.
• For the USA, Spotter's convergence at 40km radius was only 44%, compared to 78% in Europe, due to uneven distribution of PlanetLab vantage points.
• Despite strong correlation among databases, up to 20% of measurements showed errors large enough to be misleading, even at 100km radius.
• Netacuity showed better consistency in identifying correct locations when a majority vote was applied, though minority votes still pointed to incorrect or distant countries.
• Measurement-based geolocation tools like Spotter achieved high convergence (over 80% at 100km) but were still insufficient as the sole method for PoP mapping due to inherent measurement noise.
• The study confirms that geolocation databases cannot be treated as ground truth, as they contain systematic errors that span thousands of kilometers and multiple countries.