[Paper Review] Evaluating the dependence of a non-leaky intervention's partial efficacy on a categorical mark

Paul T. Edlefsen|arXiv (Cornell University)|Jun 28, 2012

Advanced Causal Inference Techniques9 references3 citations

TL;DR

This paper introduces a 'sievey-not-leaky' (SNL) framework for sieve analysis in non-leaky, partially efficacious interventions, distinguishing between efficacy due to differential protection across failure types (sieve effect) and incomplete intervention 'take' by subjects. Using Bayesian and frequentist methods, it quantifies that in the RV144 trial, ~25% of vaccine failures were due to incomplete 'take' rather than sieve effects, offering improved power to detect true sieve effects under non-leaky conditions.

ABSTRACT

We address discrete-marks survival analysis, also known as categorical sieve analysis, for a setting of a randomized placebo-controlled treatment intervention to prevent infection by a pathogen to which multiple exposures are possible, with a finite number of types of "failure". In particular, we address the case of interventions that are partially efficacious due to a combination of failure-type-dependent efficacy and subject-dependent efficacy, for an intervention that is "non-leaky" (where "leaky" interventions are those for which each exposure event has a chance of resulting in a "failure" outcome, so multiple exposures to pathogens of a single type increase the chance of failure). We introduce the notion of some-or-none interventions, which are completely effective only against some of the failure types, and are completely ineffective against the others. Under conditions of no intervention-induced failures, we introduce a framework and Bayesian and frequentist methods to detect and quantify the extent to which an intervention's partial efficacy is attributable to uneven efficacy across the failure types rather than to incomplete "take" of the intervention. These new methods provide more power than existing methods to detect sieve effects when the conditions hold. We demonstrate the new framework and methods with simulation results and new analyses of genomic signatures of HIV-1 vaccine effects in the STEP and RV144 vaccine efficacy trials.

Motivation & Objective

• To address the challenge of distinguishing between partial vaccine efficacy due to uneven protection across pathogen failure types (sieve effect) versus incomplete uptake ('take') of the intervention.
• To develop a statistical framework for non-leaky interventions—where each exposure is either fully thwarted or not, and no harm is induced—under which sieve effects can be reliably estimated.
• To provide Bayesian and frequentist methods that quantify the extent to which partial efficacy is attributable to sieve effects rather than incomplete 'take', improving detection power over existing methods.
• To apply the framework to real-world HIV-1 vaccine trials (STEP and RV144), revealing insights into immune escape and vaccine mechanism of action.

Proposed method

• Proposes a 'some-or-none' intervention model where the vaccine fully protects against some failure types (e.g., specific viral variants) and is completely ineffective against others.
• Introduces the 'sieve effect strength' parameter $ p_s $, which quantifies the proportion of partial efficacy attributable to differential protection across failure types.
• Develops a likelihood-based model under the no-harm, non-leaky assumption, where exposure always leads to infection without intervention.
• Employs both Bayesian and frequentist inference procedures, including likelihood ratio tests, to estimate $ p_s $ and test for sieve effects.
• Uses simulation studies to validate the statistical power and robustness of the methods under various scenarios, including non-zero replacement rates ($ I_E > 0 $).
• Applies the framework to genomic data from the RV144 and STEP HIV-1 vaccine trials to assess the role of specific viral envelope mutations in immune escape.

Experimental results

Research questions

• RQ1 To what extent is the partial efficacy of a non-leaky intervention attributable to differential protection across pathogen failure types (sieve effect) versus incomplete 'take' by recipients?
• RQ2 Can a statistical framework be developed to disentangle sieve effects from incomplete intervention uptake in non-leaky settings where no harm is induced?
• RQ3 How does the proposed SNL framework improve statistical power to detect true sieve effects compared to existing methods under non-leaky conditions?
• RQ4 What is the quantitative contribution of sieve effects versus incomplete 'take' to partial efficacy in the RV144 and STEP HIV-1 vaccine trials?
• RQ5 Under what conditions does the asymptotic approximation of the likelihood ratio test remain valid when $ I_E > 0 $, i.e., when replacement infections occur?

Key findings

• In the RV144 trial, approximately 25% of vaccine-recipient infections were attributable to incomplete 'take' rather than sieve effects, indicating that even with full immune targeting of all amino acids at Env 169, efficacy would have been ~50% rather than 31%.
• The sieve effect strength $ p_s $ was estimated to be around 0.75 in the RV144 trial, meaning 75% of the partial efficacy was due to differential protection against specific viral variants.
• The proposed Bayesian and frequentist methods demonstrated higher statistical power than existing approaches to detect sieve effects under non-leaky conditions.
• The likelihood ratio test based on the SNL model maintained good performance even when $ I_E > 0 $, despite asymptotic theory not strictly applying in such cases.
• In the STEP trial, the framework confirmed that although overall efficacy was not significant, the vaccine induced immune pressure on specific viral loci, indicating a potential sieve effect.
• The no-harm assumption—that each exposure would result in infection without intervention—was critical for model validity, and the framework remains applicable if the control is reinterpreted as an alternative treatment with shared would-be first failures.