[Paper Review] Measuring Transparency in Intelligent Robots

Georgios Angelopoulos, Dimitri Lacroix+4 more|arXiv (Cornell University)|Aug 29, 2024

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper introduces TOROS, the first standardized, cross-linguistically validated psychometric scale to measure perceived transparency in intelligent robots. Developed through a three-stage process with 1,223 participants across English, German, and Italian, TOROS comprises 26 items and three factors—Illegibility, Explainability, and Predictability—demonstrating high reliability, validity, and cross-linguistic consistency, enabling benchmarked research in human-robot interaction.

ABSTRACT

As robots become increasingly integrated into our daily lives, the need to make them transparent has never been more critical. Yet, despite its importance in human-robot interaction, a standardized measure of robot transparency has been missing until now. This paper addresses this gap by presenting the first comprehensive scale to measure perceived transparency in robotic systems, available in English, German, and Italian languages. Our approach conceptualizes transparency as a multidimensional construct, encompassing explainability, legibility, predictability, and meta-understanding. The proposed scale was a product of a rigorous three-stage process involving 1,223 participants. Firstly, we generated the items of our scale, secondly, we conducted an exploratory factor analysis, and thirdly, a confirmatory factor analysis served to validate the factor structure of the newly developed TOROS scale. The final scale encompasses 26 items and comprises three factors: Illegibility, Explainability, and Predictability. TOROS demonstrates high cross-linguistic reliability, inter-factor correlation, model fit, internal consistency, and convergent validity across the three cross-national samples. This empirically validated tool enables the assessment of robot transparency and contributes to the theoretical understanding of this complex construct. By offering a standardized measure, we facilitate consistent and comparable research in human-robot interaction in which TOROS can serve as a benchmark.

Motivation & Objective

• To address the lack of a standardized, empirically validated measure for perceived transparency in human-robot interaction (HRI).
• To develop a multidimensional scale capturing key aspects of robot transparency: explainability, legibility, predictability, and meta-understanding.
• To ensure cross-linguistic applicability and reliability by validating the scale in English, German, and Italian.
• To provide a benchmark tool for future HRI research assessing transparency's impact on trust, performance, and user experience.

Proposed method

• Developed the TOROS scale through a three-stage process: item generation, exploratory factor analysis (EFA), and confirmatory factor analysis (CFA).
• Collected data from 1,223 participants across three countries (Germany, Italy, and English-speaking regions) using image vignettes and video scenarios.
• Employed psychometric techniques including internal consistency (Cronbach’s alpha), model fit indices (CFI, TLI, RMSEA), and measurement invariance testing (configural, metric, scalar, residual invariance).
• Validated the scale’s convergent validity using correlations with related constructs such as trust and perceived understandability.
• Conducted cross-linguistic validation to ensure comparability across language groups, despite partial scalar invariance limitations.
• Used controlled, scenario-based stimuli to isolate and manipulate transparency levels for reliable scale calibration.

Figure 1: The three stages of the scale development.

Experimental results

Research questions

• RQ1 How can perceived transparency in intelligent robots be systematically measured as a multidimensional construct?
• RQ2 What is the underlying factor structure of perceived transparency in HRI, and how does it vary across languages?
• RQ3 To what extent is the TOROS scale reliable and valid across different linguistic and cultural contexts?
• RQ4 How do explainability, predictability, and illegibility contribute to users’ overall perception of robot transparency?
• RQ5 Can the TOROS scale serve as a valid and reliable benchmark for future HRI research on transparency-related outcomes?

Key findings

• The TOROS scale comprises 26 items and three distinct factors: Illegibility, Explainability, and Predictability, with strong internal consistency (Cronbach’s alpha > 0.80 across all language versions).
• The scale demonstrated high model fit (CFI > 0.95, TLI > 0.90, RMSEA < 0.06) in confirmatory factor analysis, confirming its robust structural validity.
• The scale achieved configural and metric measurement invariance across languages, supporting cross-cultural comparability, though scalar and residual invariance were only partially met.
• Convergent validity was confirmed through significant positive correlations with trust and perceived understandability, supporting its construct validity.
• Despite potential hindsight bias, no ceiling effect was observed in the Predictability factor, indicating the scale effectively captures variation in perceived transparency.
• Slight differences in perceived transparency across languages were observed, but these were small and did not significantly affect the overall scale performance or interaction effects.

Figure 2: Scree plot illustrating the eigenvalues of extracted factors.