[Paper Review] QB4AIRA: A Question Bank for AI Risk Assessment

Sung Une Lee, Harsha Perera+6 more|arXiv (Cornell University)|May 16, 2023

Artificial Intelligence in Healthcare and EducationMedicine3 citations

TL;DR

QB4AIRA is a curated, 293-question bank for AI risk assessment, synthesized from five global AI ethics frameworks and structured around Australia’s AI ethics principles. It enables systematic, stakeholder-agnostic risk evaluation through hierarchical categorization, decision trees, and concept mapping, supporting both manual assessments and integration into tools like an AI-powered risk chatbot.

ABSTRACT

The rapid advancement of Artificial Intelligence (AI), represented by ChatGPT, has raised concerns about responsible AI development and utilization. Existing frameworks lack a comprehensive synthesis of AI risk assessment questions. To address this, we introduce QB4AIRA, a novel question bank developed by refining questions from five globally recognized AI risk frameworks, categorized according to Australia's AI ethics principles. QB4AIRA comprises 293 prioritized questions covering a wide range of AI risk areas, facilitating effective risk assessment. It serves as a valuable resource for stakeholders in assessing and managing AI risks, while paving the way for new risk frameworks and guidelines. By promoting responsible AI practices, QB4AIRA contributes to responsible AI deployment, mitigating potential risks and harms.

Motivation & Objective

• To address the lack of a comprehensive, interconnected, and standardized question bank for AI risk assessment that spans multiple AI ethics principles and frameworks.
• To synthesize and refine risk assessment questions from five globally recognized AI risk frameworks, including NIST, EU Trustworthy AI, and Microsoft’s RAI guidelines.
• To structure the resulting questions around Australia’s AI ethics principles, ensuring alignment with widely accepted ethical standards and improving usability for diverse stakeholders.
• To evaluate QB4AIRA’s effectiveness through real-world case studies involving risk assessment sessions and a prototype smart risk assessment chatbot.
• To identify gaps in current risk assessment practices—such as lack of stage-specific questions and risk metrics—and inform future enhancements to the framework.

Proposed method

• Conducted a comparative analysis of five globally recognized AI risk assessment frameworks: NIST AI RMF, EU Assessment List for Trustworthy AI, Canada’s AIA, NSW AI Assurance Framework, and Microsoft’s RAI Impact Assessment Guide.
• Refined 382 initial questions into 293 by removing redundancies, improving clarity, and ensuring alignment with core AI ethics principles.
• Categorized questions under eight principles: Human and societal wellbeing (P1), Human-centered values (P2), Fairness (P3), Privacy and security (P4), Reliability and safety (P5), Transparency and explainability (P6), Contestability (P7), and Accountability (P8), each with a two-digit identifier.
• Applied concept mapping to identify recurring risk themes and used decision tree logic to structure questions hierarchically based on risk level and sequence.
• Developed a risk register template with fields for risk ID, category, title, description, causes, and interview questions derived from QB4AIRA for use in assessment sessions.
• Integrated QB4AIRA into a prototype smart risk assessment chatbot (SRA), leveraging a knowledge graph built from 300+ AI incident cases and using GPT-3.5 as a foundation model for response generation.

Experimental results

Research questions

• RQ1 How can a comprehensive, standardized question bank for AI risk assessment be synthesized from multiple existing global frameworks while preserving their core intent and enhancing connectivity?
• RQ2 To what extent can QB4AIRA support effective, structured, and stakeholder-agnostic AI risk assessments across diverse project stages and organizational roles?
• RQ3 What are the key limitations of current AI risk assessment practices, and how can QB4AIRA be enhanced to address them—particularly in terms of project stage alignment and risk quantification?
• RQ4 Can QB4AIRA be effectively integrated into an AI-powered tool to enable dynamic, context-aware risk assessment and improve accessibility for non-expert users?
• RQ5 What role can a principle-based, extensible question bank play in shaping future AI risk frameworks and regulatory compliance, such as with the EU AI Act or ISO standards?

Key findings

• QB4AIRA successfully consolidated 382 questions from five major AI risk frameworks into 293 prioritized, non-redundant, and clearly categorized questions aligned with Australia’s AI ethics principles.
• Case study 1 revealed that projects like PR4 and PR8 exhibited high risks in privacy, reliability, and transparency, while most projects showed low contestability risks, highlighting the need for stage-specific risk questions.
• Stakeholders identified that QB4AIRA’s current structure lacks integration with project development stages, suggesting that future versions should include stage-aware filtering to improve contextual relevance and assessment accuracy.
• The prototype Smart Risk Assessment Tool (SRA) demonstrated the feasibility of using QB4AIRA as input for a knowledge graph-based chatbot, enabling tiered risk assessment, decision tree navigation, and dynamic question selection based on user context.
• Feedback emphasized the absence of concrete risk metrics in QB4AIRA, indicating a critical gap for future development to support quantifiable risk scoring and reduce subjective decision-making in assessments.
• QB4AIRA is designed to be extensible, with plans to incorporate emerging standards such as the EU AI Act and ISO/IEC 42125, enhancing its long-term relevance and regulatory alignment.