[Paper Review] A Model for Evaluating Algorithmic Systems Accountability

Yiannis Kanellopoulos|arXiv (Cornell University)|Jul 12, 2018

Software Reliability and Analysis Research9 references4 citations

TL;DR

This paper proposes a model to evaluate accountability in algorithmic systems by assessing both the transparency of their algorithms and the organizational maturity of the institutions using them. Applied to a financial institution's classification algorithm, the model revealed partial organizational control and a lack of standardized benchmarks for validating inferencing results.

ABSTRACT

Algorithmic systems make decisions that have a great impact in our lives. As our dependency on them is growing so does the need for transparency and holding them accountable. This paper presents a model for evaluating how transparent these systems are by focusing on their algorithmic part as well as the maturity of the organizations that utilize them. We applied this model on a classification algorithm created and utilized by a large financial institution. The results of our analysis indicated that the organization was only partially in control of their algorithm and they lacked the necessary benchmark to interpret the deducted results and assess the validity of its inferencing.

Motivation & Objective

• To develop a structured model for assessing accountability in algorithmic systems.
• To examine how organizational maturity affects the transparency and reliability of algorithmic decision-making.
• To evaluate the technical and procedural transparency of a real-world classification algorithm used in finance.
• To identify gaps in benchmarking and validation processes within institutional AI deployment.
• To provide actionable insights for improving accountability in production AI systems.

Proposed method

• The model integrates technical evaluation of algorithmic components with organizational process maturity assessments.
• It evaluates algorithmic transparency through documentation, explainability, and auditability of the model's logic and training process.
• Organizational maturity is assessed via governance structures, monitoring practices, and incident response protocols.
• The model was applied to a real-world classification algorithm from a large financial institution.
• Data collection included interviews, system documentation, and analysis of model outputs and validation procedures.
• A comparative assessment was made against established standards for algorithmic accountability and model governance.

Experimental results

Research questions

• RQ1 To what extent is the algorithmic system technically transparent, allowing for audit and verification?
• RQ2 How mature are the organizational processes in managing and monitoring the algorithmic system?
• RQ3 What benchmarks exist for validating the model's inferencing and decision outcomes?
• RQ4 How well does the organization control and understand the behavior of its deployed algorithm?
• RQ5 What gaps exist in accountability mechanisms between technical design and operational practice?

Key findings

• The financial institution lacked standardized benchmarks to interpret or validate the model's inferencing results.
• The organization demonstrated only partial control over the algorithmic system, indicating weak oversight mechanisms.
• Algorithmic transparency was limited by insufficient documentation and audit trails for model decisions.
• Organizational maturity in governance and monitoring was found to be underdeveloped, especially in incident response and model versioning.
• The absence of a formal validation framework hindered the ability to assess model reliability and fairness.
• The study revealed a critical disconnect between technical capabilities and organizational accountability practices.