[Paper Review] Approximating Document Frequency with Term Count Values

Martin Klein, Michael L. Nelson|ArXiv.org|Jul 23, 2008

Natural Language Processing Techniques15 references3 citations

TL;DR

This paper demonstrates a strong correlation (Spearman's ρ ≥ 0.8, p ≤ 2.2×10⁻¹⁶) between term count (TC) and document frequency (DF) in the Web as Corpus (WaC), enabling TC to be used as a reliable proxy for DF in IDF computation. The authors validate this using Kendall’s τ and ratio analysis, concluding that TC values from large, recent corpora like Google N-grams can accurately estimate IDF for lexical signature generation.

ABSTRACT

For bounded datasets such as the TREC Web Track (WT10g) the computation of term frequency (TF) and inverse document frequency (IDF) is not difficult. However, when the corpus is the entire web, direct IDF calculation is impossible and values must instead be estimated. Most available datasets provide values for term count (TC) meaning the number of times a certain term occurs in the entire corpus. Intuitively this value is different from document frequency (DF), the number of documents (e.g., web pages) a certain term occurs in. We conduct a comparison study between TC and DF values within the Web as Corpus (WaC). We found a very strong correlation with Spearman's rho >0.8 (p<0.005) which makes us confident in claiming that for such recently created corpora the TC and DF values can be used interchangeably to compute IDF values. These results are useful for the generation of accurate lexical signatures based on the TF-IDF scheme.

Motivation & Objective

• To assess the correlation between term count (TC) and document frequency (DF) in large-scale web corpora.
• To determine whether TC can serve as a reliable proxy for DF in IDF computation when DF is unavailable.
• To evaluate the feasibility of using Google N-grams’ TC values for accurate IDF estimation despite missing DF data.
• To enable accurate TF-IDF-based lexical signature generation for web pages using TC-only corpora.

Proposed method

• Computed Spearman’s rank correlation (ρ) between TC and DF values across the WaC corpus for the top 1 million terms.
• Used Kendall’s tau (τ) to measure rank correlation, with computation time analysis to assess scalability.
• Analyzed the TC/DF ratio distribution to visualize and quantify the relationship between TC and DF.
• Compared TC frequency distributions between WaC and Google N-grams to assess corpus similarity.
• Used R-Project’s statistical functions to compute correlation coefficients on large-scale term frequency data.
• Applied data visualization (log-log plots, ratio histograms) to support empirical findings.

Experimental results

Research questions

• RQ1 Is there a significant correlation between term count (TC) and document frequency (DF) in large-scale web corpora like WaC?
• RQ2 Can TC values be used as a reliable proxy for DF in IDF computation when DF is not available?
• RQ3 How does the correlation between TC and DF in WaC compare to that in the Google N-grams corpus?
• RQ4 What is the computational feasibility of estimating DF from TC in web-scale corpora?
• RQ5 To what extent can TC-based IDF values support accurate lexical signature generation for web pages?

Key findings

• Spearman’s rank correlation (ρ) between TC and DF in the WaC corpus is ≥ 0.8 with p ≤ 2.2×10⁻¹⁶, indicating a very strong monotonic relationship.
• Kendall’s tau (τ) values range from 0.74 to 0.82, supporting a high degree of rank correlation between TC and DF.
• The mean TC/DF ratio is 1.23 with a standard deviation of 1.21, and the median ratio is 1.00, indicating tight clustering around unity.
• The TC frequency distributions of WaC and Google N-grams are highly similar, especially above the TC threshold of 200 used in N-gram construction.
• Computation of Kendall’s τ for the full WaC corpus (11M+ unique terms) is estimated at over 126 days, indicating O(n²) complexity is prohibitive for full-scale analysis.
• The strong correlation and low ratio variance support the interchangeability of TC and DF for IDF estimation in large, recent corpora.