[Paper Review] Can I Still Trust You?: Understanding the Impact of Distribution Shifts on Algorithmic Recourses.

Kaivalya Rawal, Ece Kamar+1 more|arXiv (Cornell University)|Dec 22, 2020

Explainable Artificial Intelligence (XAI)31 references13 citations

TL;DR

This paper investigates whether algorithmic recourses remain valid under distribution shifts—such as temporal, geospatial, or data correction shifts—demonstrating that state-of-the-art recourse generation methods often produce recourses that become invalid when distributions change. It establishes a theoretical lower bound on recourse invalidation probability and reveals a fundamental tradeoff between recourse validity and cost minimization.

ABSTRACT

As predictive models are being increasingly deployed to make a variety of consequential decisions ranging from hiring decisions to loan approvals, there is growing emphasis on designing algorithms that can provide reliable recourses to affected individuals. In this work, we assess the reliability of algorithmic recourses through the lens of distribution shifts i.e., we study if the recourses generated by state-of-the-art algorithms are robust to distribution shifts. To the best of our knowledge, this work makes the first attempt at addressing this critical question. We experiment with multiple synthetic and real world datasets capturing different kinds of distribution shifts including temporal shifts, geospatial shifts, and shifts due to data corrections. Our results demonstrate that all the aforementioned distribution shifts could potentially invalidate the recourses generated by state-of-the-art algorithms. Our theoretical results establish a lower bound on the probability of recourse invalidation due to distribution shifts, and show the existence of a tradeoff between this invalidation probability and typical notions of cost minimized by modern recourse generation algorithms. Our findings not only expose fundamental flaws in recourse finding strategies but also pave new way for rethinking the design and development of recourse generation algorithms.

Motivation & Objective

• To assess the robustness of algorithmic recourses under various distribution shifts, such as temporal, geospatial, and data correction shifts.
• To investigate whether recourses generated by state-of-the-art algorithms remain valid when data distributions change over time or across locations.
• To identify fundamental limitations in current recourse generation strategies that assume static data distributions.
• To establish theoretical bounds on the probability of recourse invalidation due to distribution shifts.
• To explore the tradeoff between recourse validity and the cost minimization objectives of existing recourse algorithms.

Proposed method

• The study evaluates state-of-the-art recourse generation algorithms on synthetic and real-world datasets under controlled distribution shifts.
• It simulates distribution shifts including temporal shifts (e.g., data collected at different times), geospatial shifts (e.g., data from different regions), and data correction shifts (e.g., corrected labels or features).
• Theoretical analysis derives a lower bound on the probability of recourse invalidation due to distribution shifts.
• The analysis reveals a tradeoff between this invalidation probability and the cost of recourses as defined by modern recourse algorithms.
• Empirical evaluation compares recourse validity before and after distribution shifts across multiple datasets and shift types.
• The framework evaluates both the robustness of recourses and the sensitivity of their validity to shifts in data distribution.

Experimental results

Research questions

• RQ1 To what extent do distribution shifts invalidate recourses generated by state-of-the-art recourse algorithms?
• RQ2 How do different types of distribution shifts—temporal, geospatial, and data correction—impact the validity of algorithmic recourses?
• RQ3 Is there a theoretical lower bound on the probability that a recourse will become invalid due to distribution shifts?
• RQ4 What tradeoff exists between the validity of recourses and the cost minimization objectives of recourse generation algorithms?
• RQ5 Can the robustness of recourses be improved by explicitly accounting for distribution shifts during the generation process?

Key findings

• All tested distribution shifts—temporal, geospatial, and data correction—can invalidate recourses generated by state-of-the-art algorithms.
• Theoretical analysis establishes a non-zero lower bound on the probability of recourse invalidation due to distribution shifts.
• A fundamental tradeoff exists between the probability of recourse invalidation and the cost of recourses as minimized by current algorithms.
• Even recourses that are optimal under a given distribution may fail to be valid when the underlying data distribution shifts.
• The results expose a critical flaw in current recourse generation strategies that assume static or invariant data distributions.
• The findings call for a rethinking of recourse generation algorithms to explicitly account for distribution shift robustness.