[논문 리뷰] Is explainable AI a race against model complexity?

Advait Sarkar|arXiv (Cornell University)|2022. 05. 17.

Explainable Artificial Intelligence (XAI)인용 수 9

한 줄 요약

이 논문은 설명 가능한 AI가 모델 복잡도 증가와의 경쟁이라는 본질적인 문제를 안고 있으며, 더 큰 모델은 점점 더 큰 손실 압축을 수반하는 설명이 필요하기 때문에 완전한 투명성은 비현실적임을 주장한다. 설명이 모델 복잡도에 따라 따라오지 못할지라도, 이 논문은 상호작용 설계, 메타모델, 사용자 권한 부여와 같은 실용적이고 인간 중심의 전략을 제안하여, 본질적으로 투명하지 않은 상황에서도 신뢰를 유지할 수 있는 '해석 불가능성 이후의 세계'를 탐색한다.

ABSTRACT

Explaining the behaviour of intelligent systems will get increasingly and perhaps intractably challenging as models grow in size and complexity. We may not be able to expect an explanation for every prediction made by a brain-scale model, nor can we expect explanations to remain objective or apolitical. Our functionalist understanding of these models is of less advantage than we might assume. Models precede explanations, and can be useful even when both model and explanation are incorrect. Explainability may never win the race against complexity, but this is less problematic than it seems.

연구 동기 및 목표

• growing complexity of AI models makes explainability inherently unattainable.
• To analyze the limitations of current explanation methods as models scale toward brain-scale complexity.
• To explore how human reasoning and cognitive constraints inform the design of effective AI explanations.
• To evaluate the feasibility and shortcomings of legal, technical, and interactive approaches to AI explainability.
• To propose a shift from seeking perfect explanations to designing systems that support user agency and practical trust in high-stakes AI applications.

제안 방법

• Analyzes the theoretical and practical barriers to explaining large-scale neural networks, focusing on lossy compression in explanations.
• Draws analogies between human explanation processes and AI explanation, emphasizing cognitive limits and the role of abstraction.
• Reviews existing explanation techniques such as saliency maps, counterfactuals, and attention visualization in image and NLP tasks.
• Examines legislative frameworks like the GDPR’s 'right to explanation' and their conceptual and practical shortcomings.
• Proposes alternative strategies, including metamodels for explanation, end-user programming, and interactive design, to empower users.
• Applies the Kübler-Ross model of grief to frame societal and user responses to AI opacity, suggesting that acceptance and adaptation are more realistic goals than full explainability.

실험 결과

연구 질문

• RQ1 Can explanations of increasingly complex AI models remain meaningful, accurate, and useful as model size grows?
• RQ2 To what extent do cognitive and perceptual limits constrain the effectiveness of AI explanations?
• RQ3 How do current legal frameworks like the GDPR’s 'right to explanation' address or fail to address the core challenges of AI explainability?
• RQ4 What alternative strategies can be employed when perfect explanation is infeasible due to model complexity?
• RQ5 How can users be empowered to interact with and shape AI systems in the absence of full transparency?

주요 결과

• As model complexity increases, explanations must perform greater lossy compression, leading to inevitable loss of detail and nuance in the explanation process.
• Current explanation methods such as saliency maps and attention visualization are forms of lossy compression and cannot fully represent the decision-making process of large models.
• The 'right to explanation' in the GDPR is conceptually bold but practically weak, as it does not guarantee meaningful or actionable explanations.
• Legal and technical approaches to explanation often fail to address the root issue: the fundamental incompatibility between model complexity and human cognitive limits.
• Pragmatic alternatives—such as interactive design, end-user programming, and metamodels—offer more viable paths forward than pursuing perfect explanation.
• The paper concludes that while explainable AI may never 'win' the race against complexity, society can adapt through user empowerment and post-explainability strategies.