[Paper Review] Challenging the Human-in-the-loop in Algorithmic Decision-making

Sebastian Tschiatschek, Eugenia Stamboliev+3 more|arXiv (Cornell University)|May 17, 2024

Simulation Techniques and ApplicationsDecision Sciences3 citations

TL;DR

This paper challenges the assumption that human-in-the-loop (HIL) oversight in algorithmic decision-making (ADM) inherently ensures ethical or effective outcomes. It demonstrates that practical decision-makers (PDMs), who make final decisions based on algorithmic recommendations, can unintentionally undermine strategic decision-makers' (SDMs) societal goals due to value misalignment and unmet information needs—highlighting the PDM's critical, often underappreciated, role as a political and ethical actor in ADM systems.

ABSTRACT

We discuss the role of humans in algorithmic decision-making (ADM) for socially relevant problems from a technical and philosophical perspective. In particular, we illustrate tensions arising from diverse expectations, values, and constraints by and on the humans involved. To this end, we assume that a strategic decision-maker (SDM) introduces ADM to optimize strategic and societal goals while the algorithms' recommended actions are overseen by a practical decision-maker (PDM) - a specific human-in-the-loop - who makes the final decisions. While the PDM is typically assumed to be a corrective, it can counteract the realization of the SDM's desired goals and societal values not least because of a misalignment of these values and unmet information needs of the PDM. This has significant implications for the distribution of power between the stakeholders in ADM, their constraints, and information needs. In particular, we emphasize the overseeing PDM's role as a potential political and ethical decision maker, who acts expected to balance strategic, value-driven objectives and on-the-ground individual decisions and constraints. We demonstrate empirically, on a machine learning benchmark dataset, the significant impact an overseeing PDM's decisions can have even if the PDM is constrained to performing only a limited amount of actions differing from the algorithms' recommendations. To ensure that the SDM's intended values are realized, the PDM needs to be provided with appropriate information conveyed through tailored explanations and its role must be characterized clearly. Our findings emphasize the need for an in-depth discussion of the role and power of the PDM and challenge the often-taken view that just including a human-in-the-loop in ADM ensures the 'correct' and 'ethical' functioning of the system.

Motivation & Objective

• To critically examine the role of practical decision-makers (PDMs) in algorithmic decision-making (ADM) systems, particularly in public services.
• To investigate how value misalignment and unmet information needs between strategic decision-makers (SDMs) and PDMs can undermine societal goals in ADM.
• To challenge the assumption that human oversight alone ensures ethical or correct functioning of ADM systems.
• To emphasize the need for tailored, value-informed explanations that address the distinct information needs of SDMs and PDMs.
• To advocate for a rethinking of power distribution and role clarity between SDMs and PDMs in ADM governance frameworks.

Proposed method

• The study conceptualizes a two-tiered ADM framework: strategic decision-makers (SDMs) set long-term goals, while practical decision-makers (PDMs) make final, individual-level decisions based on algorithmic recommendations.
• It introduces a theoretical and empirical analysis of tensions arising from divergent values, constraints, and information needs between SDMs and PDMs.
• The authors conduct experiments on a machine learning benchmark dataset to simulate how PDMs' limited deviations from algorithmic recommendations can significantly alter system outcomes.
• The method emphasizes iterative, interactive, and emergent explanation design tailored to different stakeholder roles—particularly focusing on value alignment and ethical decision-making.
• It proposes a framework for value-informed explanations that account for PDMs' implicit strategic role and their potential to reshape ADM outcomes.
• The approach includes modeling PDM behavior through observed decisions and comparing them to intended SDM values to assess alignment and risk.

Experimental results

Research questions

• RQ1 How do value misalignments between strategic decision-makers (SDMs) and practical decision-makers (PDMs) affect the realization of societal goals in algorithmic decision-making?
• RQ2 To what extent can a PDM’s limited but deliberate deviation from algorithmic recommendations alter the overall behavior and outcomes of an ADM system?
• RQ3 What are the distinct information needs of SDMs and PDMs in ADM, and how should explanations be tailored to meet them?
• RQ4 In what ways does the PDM’s role function as a de facto political and ethical decision-maker, despite being positioned as a corrective oversight actor?
• RQ5 How can explanations be designed to make the PDM’s implicit strategic influence visible and accountable within ADM systems?

Key findings

• The PDM’s decisions can significantly alter the behavior of an ADM system, even when they deviate only minimally from algorithmic recommendations, as demonstrated in experiments on a benchmark dataset.
• Value misalignment between SDMs and PDMs can lead to a transfer of power from the SDM to the PDM, undermining the intended societal goals of the ADM system.
• Unmet information needs of the PDM—particularly around societal values and system dynamics—can result in decisions that counteract the SDM’s strategic objectives.
• The PDM functions not just as a corrective but as a strategic and ethical decision-maker, whose role must be formally recognized and supported with tailored explanations.
• Current XAI approaches often fail to address the PDM’s need to understand and act upon societal values, highlighting a critical gap in explanation design.
• The study suggests that explanations should explicitly frame the PDM’s role as a strategic actor and include mechanisms for value inference and alignment checks between PDM decisions and SDM intentions.