[Paper Review] Towards a multi-stakeholder value-based assessment framework for algorithmic systems

Mireia Yurrita, Dave Murray-Rust+2 more|arXiv (Cornell University)|Jan 1, 2022

Ethics and Social Impacts of AI193 references32 citations

TL;DR

This paper proposes a multi-stakeholder, value-based assessment framework for algorithmic systems that extends beyond bias auditing to include 11 core ethical values arranged in a circular, bipolar structure to visualize trade-offs and tensions. It operationalizes these values through quantifiable indicators, process-oriented practices, and signifiers, and maps tailored communication means to diverse stakeholders, enabling inclusive, context-aware ethical evaluation throughout the ML lifecycle.

ABSTRACT

<p>In an effort to regulate Machine Learning-driven (ML) systems, current auditing processes mostly focus on detecting harmful algorithmic biases. While these strategies have proven to be impactful, some values outlined in documents dealing with ethics in ML-driven systems are still underrepresented in auditing processes. Such unaddressed values mainly deal with contextual factors that cannot be easily quantified. In this paper, we develop a value-based assessment framework that is not limited to bias auditing and that covers prominent ethical principles for algorithmic systems. Our framework presents a circular arrangement of values with two bipolar dimensions that make common motivations and potential tensions explicit. In order to operationalize these high-level principles, values are then broken down into specific criteria and their manifestations. However, some of these value-specific criteria are mutually exclusive and require negotiation. As opposed to some other auditing frameworks that merely rely on ML researchers' and practitioners' input, we argue that it is necessary to include stakeholders that present diverse standpoints to systematically negotiate and consolidate value and criteria tensions. To that end, we map stakeholders with different insight needs, and assign tailored means for communicating value manifestations to them. We, therefore, contribute to current ML auditing practices with an assessment framework that visualizes closeness and tensions between values and we give guidelines on how to operationalize them, while opening up the evaluation and deliberation process to a wide range of stakeholders.</p>

Motivation & Objective

• Address the gap in current ML auditing practices that focus narrowly on bias detection while neglecting broader ethical values such as contestability, transparency, and accountability.
• Develop a structured, actionable framework that operationalizes high-level ethical principles into concrete criteria and manifestations for practical implementation.
• Facilitate inclusive ethical deliberation by identifying and engaging diverse stakeholders with tailored communication methods for value manifestations.
• Visualize value tensions and common motivations through a circular, bipolar arrangement of ethical values to support proactive ethical intervention in ML development.
• Provide a flexible, extensible framework that supports context-specific adaptations and encourages ongoing stakeholder engagement throughout the ML pipeline.

Proposed method

• Design a circular, bipolar framework organizing 11 prominent ethical values (e.g., fairness, transparency, contestability) to make value interactions, trade-offs, and shared motivations explicit.
• Break down each value into specific criteria and their manifestations, categorized as quantifiable indicators, process-oriented practices, or signifiers (perceived cues of system affordances).
• Map stakeholder profiles (e.g., developers, end-users, auditors, decision subjects) to their distinct insight needs and assign tailored communication means (e.g., dashboards, reports, visualizations) for value manifestations.
• Integrate the framework into the ML development and deployment pipeline to support both retrospective assessment and proactive value integration during design.
• Leverage existing tools and practices (e.g., Values Dashboard, GitHub-style timelines) to support stakeholder engagement and iterative evaluation across development phases.
• Host the framework in an open, collaborative online repository to enable community contributions and continuous extension of tools and communication methods.

Experimental results

Research questions

• RQ1 How can ethical values in ML systems be systematically organized to reveal both shared motivations and inherent tensions?
• RQ2 What criteria and manifestations can operationalize high-level ethical values such as fairness, transparency, and contestability in a context-aware manner?
• RQ3 How can diverse stakeholders be meaningfully engaged in the assessment of algorithmic systems through tailored communication of value manifestations?
• RQ4 What role do contextual factors and long-term system impacts play in ethical assessment, and how can they be integrated into auditing frameworks?
• RQ5 How can the framework support both retrospective evaluation and proactive ethical design in the ML lifecycle?

Key findings

• The circular, bipolar arrangement of 11 ethical values enables the visualization of both shared motivations and trade-offs, making value tensions and interdependencies explicit.
• Each value is decomposed into actionable criteria manifested as quantifiable indicators, process-oriented practices, or signifiers, enabling practical implementation across the ML pipeline.
• Stakeholders with varying levels of technical expertise and insight needs are mapped to specific communication means, such as dashboards for developers and simplified reports for end-users.
• The framework identifies research gaps, such as the scarcity of communication tools for fairness manifestations targeting decision subjects, highlighting underdeveloped but critical areas for future work.
• The framework supports both retrospective assessment and proactive ethical design, with the potential to integrate into existing development workflows through tools like the Values Dashboard.
• The open, community-driven repository model encourages ongoing extension and adaptation of the framework, fostering long-term sustainability and context-specific customization.