[Paper Review] Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI

Alon Jacovi, Ana Marasović+2 more|arXiv (Cornell University)|Oct 15, 2020

Explainable Artificial Intelligence (XAI)71 references66 citations

TL;DR

The paper formalizes Human-AI trust inspired by interpersonal trust, introduces contractual trust and trustworthiness, and distinguishes warranted versus unwarranted trust, linking trust to XAI and evaluation.

ABSTRACT

Trust is a central component of the interaction between people and AI, in that 'incorrect' levels of trust may cause misuse, abuse or disuse of the technology. But what, precisely, is the nature of trust in AI? What are the prerequisites and goals of the cognitive mechanism of trust, and how can we promote them, or assess whether they are being satisfied in a given interaction? This work aims to answer these questions. We discuss a model of trust inspired by, but not identical to, sociology's interpersonal trust (i.e., trust between people). This model rests on two key properties of the vulnerability of the user and the ability to anticipate the impact of the AI model's decisions. We incorporate a formalization of 'contractual trust', such that trust between a user and an AI is trust that some implicit or explicit contract will hold, and a formalization of 'trustworthiness' (which detaches from the notion of trustworthiness in sociology), and with it concepts of 'warranted' and 'unwarranted' trust. We then present the possible causes of warranted trust as intrinsic reasoning and extrinsic behavior, and discuss how to design trustworthy AI, how to evaluate whether trust has manifested, and whether it is warranted. Finally, we elucidate the connection between trust and XAI using our formalization.

Motivation & Objective

• Define trust between a human and an AI model inspired by sociological interpersonal trust.
• Introduce contractual trust and distinguish trustworthiness.
• Differentiate warranted and unwarranted trust and discuss their implications.
• Explain how trust relates to XAI and model evaluation.
• Propose a framework for designing and assessing trustworthy AI in real-world interactions.

Proposed method

• Adopt two core properties of trust: user vulnerability and the ability to anticipate the AI’s impact.
• Formalize contractual trust to specify what the AI is trusted to do.
• Differentiate trustworthiness from trust and define warranted vs unwarranted trust.
• Characterize intrinsic trust (aligned internal reasoning) and extrinsic trust (credible external behavior) as trust-promoting mechanisms.
• Relate the framework to European guidelines and standard documentations (e.g., datasheets, model cards) to specify contracts.
• Discuss evaluation methodologies (proxy explanations, post-deployment data, test sets) to justify extrinsic trust.

Experimental results

Research questions

• RQ1 What are the prerequisites for Human-AI trust?
• RQ2 How can contractual trust and trustworthiness be formally defined for Human-AI interactions?
• RQ3 What differentiates warranted from unwarranted trust in AI?
• RQ4 How can trust be promoted and evaluated in practice, including the role of XAI?
• RQ5 How does the proposed formalization connect to existing guidelines and documentation practices?

Key findings

• Trust between a human and AI is a directional transaction requiring vulnerability and anticipation of impact.
• Contractual trust and trustworthiness can be formalized to distinguish when trust is warranted versus unwarranted.
• Trust can be context-dependent, with contracts conditioned on interaction context.
• Trust mechanisms include intrinsic trust (explainable reasoning aligned with user priors) and extrinsic trust (trust in evaluation methods and data).
• Evaluation schemes (proxy judgments, post-deployment data, and test sets) are essential to establishing extrinsic trust and to assess whether a contract can be maintained.
• Explainability in AI should align with user priors and contract-specific goals to foster genuine trust rather than mere perception.