[Paper Review] Investigating the Change of Web Pages' Titles Over Time

Martin Klein, Michael L. Nelson|ArXiv.org|Jul 20, 2009

Web Data Mining and Analysis23 references3 citations

TL;DR

This paper investigates the stability of web page titles over time using Internet Archive snapshots to assess their viability as low-cost proxies for lexical signatures (LS) in rediscovering missing web pages. It finds that titles change infrequently and minimally—evidenced by low change frequency and high Levenshtein similarity—suggesting titles are highly stable and thus effective for real-time search-based recovery of 404 pages.

ABSTRACT

Inaccessible web pages are part of the browsing experience. The content of these pages however is often not completely lost but rather missing. Lexical signatures (LS) generated from the web pages' textual content have been shown to be suitable as search engine queries when trying to discover a (missing) web page. Since LSs are expensive to generate, we investigate the potential of web pages' titles as they are available at a lower cost. We present the results from studying the change of titles over time. We take titles from copies provided by the Internet Archive of randomly sampled web pages and show the frequency of change as well as the degree of change in terms of the Levenshtein score. We found very low frequencies of change and high Levenshtein scores indicating that titles, on average, change little from their original, first observed values (rooted comparison) and even less from the values of their previous observation (sliding).

Motivation & Objective

• To evaluate the stability of web page titles over time as a low-cost alternative to generating lexical signatures (LS) for rediscovering missing web pages.
• To determine whether titles remain sufficiently consistent to serve as reliable search queries when a web page returns a 404 error or unrelated content.
• To assess the feasibility of using title-based queries in real-time systems for missing page recovery, reducing computational cost compared to LS generation.
• To explore the potential of titles as early-stage queries in a multi-stage discovery pipeline before resorting to more complex LS generation from cached content or link neighborhoods.

Proposed method

• Collected web page titles from the Internet Archive (IA) for a random sample of web pages, capturing multiple snapshots over time.
• Computed title change frequency by comparing each title to its first observed version (rooted comparison) and to its immediately preceding version (sliding comparison).
• Measured the degree of title change using the Levenshtein edit distance score, which quantifies the number of single-character edits needed to transform one string into another.
• Analyzed the distribution of Levenshtein scores across all titles to assess average similarity and change severity.
• Used the Levenshtein score to quantify both the frequency and magnitude of title changes, with higher scores indicating greater similarity and less change.
• Evaluated title stability under two comparison models: rooted (vs. first observed title) and sliding (vs. previous observed title), to understand temporal evolution.

Experimental results

Research questions

• RQ1 How frequently do web page titles change over time, as observed across multiple snapshots from the Internet Archive?
• RQ2 To what extent do web page titles change in terms of lexical content, as measured by Levenshtein edit distance?
• RQ3 How does title stability differ when comparing titles to their initial version (rooted) versus their immediately preceding version (sliding)?
• RQ4 Can web page titles be reliably used as search engine queries to rediscover missing or moved web pages, given their stability over time?

Key findings

• The frequency of title changes is very low, with the majority of titles remaining unchanged across multiple snapshots over time.
• Levenshtein scores for title comparisons are consistently high—indicating minimal lexical change—across both rooted and sliding comparisons.
• On average, titles show significantly less change from their previous observed version (sliding comparison) than from their original version (rooted comparison), suggesting titles stabilize after initial changes.
• The high Levenshtein scores imply that titles are highly stable and retain their original meaning and structure over time, making them reliable for search-based discovery.
• The results support the use of titles as a low-cost, effective alternative to lexical signatures (LS) for querying search engines to recover missing web pages.
• The stability of titles suggests they can be used in real-time systems to quickly offer relevant results when a 404 error occurs, before more expensive LS computation is required.