[Paper Review] Document Retrieval for Large Scale Content Analysis using Contextualized Dictionaries

Gregor Wiedemann, Andreas Niekler|arXiv (Cornell University)|Jul 11, 2017

Advanced Text Analysis Techniques14 references6 citations

TL;DR

This paper proposes a contextualized dictionary approach for document retrieval in large-scale content analysis, using topic models and co-occurrence data to generate weighted term dictionaries from paradigmatic reference documents. The method improves retrieval performance over baseline term weighting by incorporating semantic context, achieving a MAP of 0.861 with topic model-based dictionaries and co-occurrence matching.

ABSTRACT

This paper presents a procedure to retrieve subsets of relevant documents from large text collections for Content Analysis, e.g. in social sciences. Document retrieval for this purpose needs to take account of the fact that analysts often cannot describe their research objective with a small set of key terms, especially when dealing with theoretical or rather abstract research interests. Instead, it is much easier to define a set of paradigmatic documents which reflect topics of interest as well as targeted manner of speech. Thus, in contrast to classic information retrieval tasks we employ manually compiled collections of reference documents to compose large queries of several hundred key terms, called dictionaries. We extract dictionaries via Topic Models and also use co-occurrence data from reference collections. Evaluations show that the procedure improves retrieval results for this purpose compared to alternative methods of key term extraction as well as neglecting co-occurrence data.

Motivation & Objective

• To address the challenge of retrieving relevant documents from large text collections when research interests are abstract and cannot be captured by simple keyword queries.
• To develop a retrieval method that leverages manually curated reference documents as a basis for query formulation, rather than relying on isolated key terms.
• To improve retrieval performance by integrating contextual information through co-occurrence data and topic model-based dictionary weighting.
• To evaluate the effectiveness of this approach in comparison to standard term weighting methods like tf/idf and unigram matching.
• To demonstrate the method’s utility in social science content analysis, particularly for identifying documents with specific ideological or discursive features.

Proposed method

• The method begins with a semi-supervised dictionary extraction process using Latent Dirichlet Allocation (LDA) on a collection of paradigmatic reference documents to identify topic-relevant terms.
• Term weights in the dictionary are computed as the sum of probabilities across all topics, reflecting each term’s contribution to the semantic structure of the reference collection.
• Co-occurrence data is extracted from both the reference collection and a generic language corpus to model contextual relationships between terms.
• Document relevance is scored using a hybrid similarity measure that combines unigram matching and context-based similarity, weighted by a parameter α.
• The Condorcet method is applied to rank candidate documents based on pairwise comparisons across multiple retrieval systems, selecting a robust set of pseudorelevant documents for evaluation.
• Performance is evaluated using mean average precision (MAP) and precision at k, with manual annotation of relevance for top-ranked documents.

Experimental results

Research questions

• RQ1 Can a dictionary derived from topic models and co-occurrence data improve document retrieval performance for abstract, theory-laden research interests in content analysis?
• RQ2 How does the inclusion of co-occurrence data affect retrieval quality compared to using raw term frequencies alone?
• RQ3 Does a topic model-based dictionary outperform a tf/idf-based dictionary when used in retrieval for content analysis tasks?
• RQ4 To what extent is the retrieval performance robust across different ranking positions, especially in the lower ranks of the result list?
• RQ5 Can a consensus ranking via the Condorcet method effectively identify a reliable set of pseudorelevant documents for evaluation in the absence of gold-standard relevance labels?

Key findings

• The best-performing retrieval system, which combines unigram and co-occurrence matching with α=14, achieved a mean average precision (MAP) of 0.861 using topic model-based dictionaries.
• Systems using topic model-based dictionaries outperformed those using tf/idf-based dictionaries when co-occurrence data was included, with a MAP of 0.861 vs. 0.823.
• The precision at k evaluation showed that the top 10 documents of the best system had a precision of 1.0, while the precision remained high (0.9) at rank 501, indicating strong performance in lower ranks.
• The use of co-occurrence data significantly improved retrieval quality, as systems relying solely on unigram matching had lower precision and MAP values.
• The Condorcet-based selection of pseudorelevant documents yielded a stable set of 54 documents, with robust ranking across multiple systems, validating the evaluation methodology.
• The method demonstrated that contextualized dictionaries derived from topic models and co-occurrence data are effective for retrieving documents aligned with abstract or theoretical research interests in content analysis.