[Paper Review] Likelihood Ratio as Weight of Forensic Evidence: A Metrological Perspective

Steven P. Lund, Hari Iyer|arXiv (Cornell University)|Aug 26, 2016

Time Series Analysis and Forecasting14 references3 citations

TL;DR

This paper challenges the normative use of the likelihood ratio (LR) as a transferable measure of forensic evidence weight, arguing that LRs—being inherently subjective—lack metrological rigor without comprehensive uncertainty characterization. It proposes that forensic experts should instead present transparent, assumption-laden analyses using tools like the Lattice of Assumptions and Uncertainty Pyramid to support decision-makers in forming their own interpretations, rather than claiming objective measurement status for the LR.

ABSTRACT

In this article we provide a rebuttal against the possible perception that a single number, such as the Likelihood Ratio, can provide an objective, authoritative or definitive weight of evidence. We also illustrate the extent to which conclusions can vary depending on the assumptions used in the analysis, even under alternative assumptions that are judged to be consistent with available empirical information. To facilitate these goals, we introduce the notion of a Lattice of Assumptions and an Uncertainty Pyramid illustrated in the context of a previously published example involving glass evidence. We take the position that rather than focusing on a single number summary as the weight of evidence it is the duty of the forensic expert to assist the trier of fact in forming their own interpretations from a clear understanding of the objective and demonstrably available information. We hope the presented arguments will inspire those in the forensic science community to pursue establishing their practice on a solid foundation of measurement science.

Motivation & Objective

• To challenge the prevailing normative view that forensic experts should report a single likelihood ratio (LR) as the 'weight of evidence' for decision-makers.
• To highlight that LRs, as subjective Bayesian constructs, lack metrological properties such as traceability, repeatability, and reproducibility required for measurement.
• To argue that presenting an LR without uncertainty characterization misrepresents it as a universal, precise measurement, undermining its transferability.
• To propose that forensic experts should instead support decision-makers by clearly communicating the assumptions and uncertainties underlying their interpretations.
• To encourage the forensic science community to adopt measurement science principles in expert reporting, emphasizing transparency over claimed objectivity.

Proposed method

• Adopt a metrological perspective to evaluate whether the likelihood ratio (LR) qualifies as a measurement of evidence weight.
• Introduce the 'Lattice of Assumptions' to systematically explore the space of plausible modeling choices that are consistent with empirical data.
• Construct an 'Uncertainty Pyramid' to visualize the range of LR values obtainable under different plausible assumptions, thereby quantifying uncertainty in interpretation.
• Use Bayesian updating as a framework to illustrate the disconnect between expert-provided LR and its use by decision-makers (DMs), emphasizing that the LR is inherently personal.
• Advocate for reporting procedures and outcomes transparently, including data, assumptions, and modeling choices, to enable DMs to form their own posterior odds.
• Suggest alternative use of LR as a discriminant score, evaluated empirically via ROC curves, if probabilistic interpretation is not defensible due to uncertainty.

Experimental results

Research questions

• RQ1 Can a single likelihood ratio (LR) value be considered a valid measurement of the weight of forensic evidence under metrological standards?
• RQ2 To what extent do subjective modeling choices—such as population definition, likelihood functions, and priors—affect the LR and undermine its reproducibility?
• RQ3 What uncertainty characterization is required for an LR to be considered a reliable, transferable measure in forensic decision-making?
• RQ4 Under what conditions is it inappropriate or misleading to present an LR as a definitive weight of evidence?
• RQ5 How can forensic experts better support decision-makers without claiming objectivity for inherently subjective interpretations?

Key findings

• The likelihood ratio (LR) as currently reported by forensic experts lacks metrological status because it is not traceable to a standard, lacks calibration, and is not accompanied by uncertainty estimates.
• The LR is fundamentally a personal probability ratio, not a universal measurement, and treating it as such misrepresents its role in Bayesian updating.
• Reproducibility of LR values across experts or decision-makers is highly questionable due to dependence on subjective assumptions, which are rarely quantified.
• Uncertainty in LR values can be explored through systematic analysis of plausible assumptions using tools like the Lattice of Assumptions and Uncertainty Pyramid.
• When uncertainty characterization is infeasible, the authors argue that the LR should not be reported as a weight of evidence, and alternative reporting methods should be used.
• Empirical evaluation of LR as a discriminant score via ROC curves may be a more defensible alternative to probabilistic interpretation when uncertainty is too high to quantify.