[Paper Review] When Index Term Probability Violates the Classical Probability Axioms Quantum Probability can be a Necessary Theory for Information Retrieval

Massimo Melucci|arXiv (Cornell University)|Mar 12, 2012

Logic, Reasoning, and Knowledge7 references3 citations

TL;DR

This paper argues that when index term probabilities in information retrieval violate classical probability axioms—particularly the Law of Total Probability—quantum probability (QP) becomes a necessary theoretical framework. It demonstrates that violations of statistical invariants indicate incompatible event spaces, rendering classical Bayesian models inapplicable, and proposes QP as a more general, non-classical alternative capable of modeling contextual, non-commuting relevance relationships in IR systems.

ABSTRACT

Probabilistic models require the notion of event space for defining a probability measure. An event space has a probability measure which ensues the Kolmogorov axioms. However, the probabilities observed from distinct sources, such as that of relevance of documents, may not admit a single event space thus causing some issues. In this article, some results are introduced for ensuring whether the observed prob- abilities of relevance of documents admit a single event space. More- over, an alternative framework of probability is introduced, thus chal- lenging the use of classical probability for ranking documents. Some reflections on the convenience of extending the classical probabilis- tic retrieval toward a more general framework which encompasses the issues are made.

Motivation & Objective

• To investigate whether observed conditional probabilities in information retrieval can be modeled within a single classical event space.
• To identify when violations of the Law of Total Probability (LTP) occur due to incompatible event spaces.
• To argue that classical probabilistic models fail when LTP is violated, necessitating a more general framework.
• To propose quantum probability (QP) as a necessary alternative for modeling non-classical, context-dependent relevance relationships in IR.
• To provide theoretical justification for using Hilbert spaces and QP in IR, grounded in empirical violations of classical probability axioms.

Proposed method

• Introduces a statistical invariant (inequality 4) as a necessary and sufficient condition for the existence of a single event space.
• Uses the inequality $\left\|\frac{P_{2}(B)-P_{1}(B|A)}{P_{4}(B|\bar{A})-P_{1}(B|A)}\right\|\leq 1$ to test whether observed probabilities can coexist in one probability space.
• Applies the violation of this invariant as evidence that multiple, incompatible event spaces are used in practice.
• Proposes that quantum probability (QP) provides a non-classical mathematical framework capable of modeling such non-commuting, context-sensitive events.
• Models documents and relevance as vectors in a complex Hilbert space, where probabilities are derived from inner products (Born rule).
• Extends the probabilistic retrieval model to use QP-based probability amplitudes, allowing for interference and contextual dependencies not captured by classical models.

Experimental results

Research questions

• RQ1 Under what conditions do observed conditional probabilities in IR fail to admit a single classical event space?
• RQ2 Can violations of the Law of Total Probability (LTP) be empirically detected and quantified in retrieval systems?
• RQ3 Why is classical Bayesian probability insufficient when multiple, incompatible event spaces are involved in relevance estimation?
• RQ4 Is quantum probability a necessary framework for modeling information retrieval when classical probability axioms are violated?
• RQ5 How can a non-classical probability model like QP better represent contextual and non-commuting relevance relationships in IR?

Key findings

• Violations of the statistical invariant (inequality 4) indicate that observed probabilities cannot be derived from a single event space, invalidating classical Bayesian reasoning.
• Empirical experiments show that terms violating the LTP lead to significant improvements in Mean Average Precision (MAP), contradicting classical intuition.
• The scatterplot in Figure 1 demonstrates a strong correlation between the degree of LTP violation and increased retrieval effectiveness.
• When probabilities are estimated from distinct event spaces, classical properties like Bayes' Theorem and distributivity no longer hold.
• Quantum probability provides a mathematically consistent framework for modeling non-commuting, context-dependent relevance events that classical probability cannot.
• Theoretical results, including Corollary 4.1, confirm that even if a full set of probabilities violates a single space, subsets may still be consistently modeled, supporting the need for a generalized probability framework.