[Paper Review] Human computation requires and enables a new approach to ethical review

Libuše Hannah Vepřek, Patricia Seymour+1 more|arXiv (Cornell University)|Nov 21, 2020

Ethics and Social Impacts of AI19 references4 citations

TL;DR

This paper proposes a dynamic, crowdsourced ethical review framework for human computation systems, leveraging participatory design and a sandboxed technosocial platform to enable real-time ethical evaluation and continuous evolution of ethical standards. It introduces a living ethics model that integrates IRB review with community input, ensuring adaptability to emerging human-computer collaboration contexts while enhancing transparency and reproducibility in ethical oversight.

ABSTRACT

With humans increasingly serving as computational elements in distributed information processing systems and in consideration of the profit-driven motives and potential inequities that might accompany the emerging thinking economy[1], we recognize the need for establishing a set of related ethics to ensure the fair treatment and wellbeing of online cognitive laborers and the conscientious use of the capabilities to which they contribute. Toward this end, we first describe human-in-the-loop computing in context of the new concerns it raises that are not addressed by traditional ethical research standards. We then describe shortcomings in the traditional approach to ethical review and introduce a dynamic approach for sustaining an ethical framework that can continue to evolve within the rapidly shifting context of disruptive new technologies.

Motivation & Objective

• To address the inadequacy of traditional IRB standards in evaluating human computation and citizen science projects.
• To develop a living ethics framework that evolves with technological and social changes in human-in-the-loop systems.
• To integrate ethical review into the design process via a transparent, participatory, and reproducible platform.
• To ensure fair treatment and wellbeing of online cognitive laborers in distributed computation systems.
• To enable ethical oversight that reflects the complex, dual roles of participants as both contributors and researchers.

Proposed method

• Design a technosocial platform with a sandbox environment where researchers can simulate experiments and IRB reviewers can observe them from a participant’s perspective.
• Integrate real-time, interactive IRB review by allowing experts to comment and suggest edits directly on the interface.
• Log all ethical issues, dialogues, and solutions in a transparent, version-controlled repository linked to the experiment’s snapshot.
• Enable community-driven review by recruiting non-expert contributors when ethical guidelines are ambiguous or insufficient.
• Use community feedback to iteratively update and refine the ethical guidelines, ensuring they evolve with the field.
• Embed reproducibility into the ethical review process by preserving full audit trails of decisions and revisions.

Experimental results

Research questions

• RQ1 How can ethical review processes be adapted to the dynamic, participatory nature of human computation and citizen science?
• RQ2 What mechanisms can ensure ethical oversight remains transparent, reproducible, and responsive to emerging technological contexts?
• RQ3 How can the roles of participants—as both contributors and potential researchers—be ethically acknowledged and protected?
• RQ4 In what ways can community input improve the relevance and adaptability of ethical standards in human-computer collaboration?
• RQ5 How can ethical review be made less burdensome and more effective through immersive, interactive evaluation?

Key findings

• The proposed platform enables IRB reviewers to evaluate experiments by experiencing them firsthand in a sandbox, reducing reliance on abstract descriptions and improving accuracy.
• Ethical issues and dialogues are fully logged and versioned, creating a reproducible audit trail that enhances transparency and accountability.
• Community contributors can be recruited to review ambiguous ethical cases, mimicking academic peer review and increasing contextual relevance.
• Ethical guidelines are not static; they can be amended based on real-world review outcomes, enabling continuous evolution of the ethical framework.
• The integration of participatory methods from human computation into ethical review allows for scalable, adaptive, and inclusive governance of AI-human systems.
• The system addresses key limitations of traditional IRB by reducing review time, increasing participant empathy, and aligning ethical standards with actual user experiences.