[Paper Review] MAILS -- Meta AI Literacy Scale: Development and Testing of an AI Literacy Questionnaire Based on Well-Founded Competency Models and Psychological Change- and Meta-Competencies

Astrid Carolus, Martin Koch+3 more|arXiv (Cornell University)|Feb 18, 2023

Ethics and Social Impacts of AI12 citations

TL;DR

This paper develops and validates the MAILS questionnaire to assess AI literacy, combining standard AI-literacy facets with psychological change- and meta-competencies, using 300 German-speaking adults and confirmatory factor analysis.

ABSTRACT

The goal of the present paper is to develop and validate a questionnaire to assess AI literacy. In particular, the questionnaire should be deeply grounded in the existing literature on AI literacy, should be modular (i.e., including different facets that can be used independently of each other) to be flexibly applicable in professional life depending on the goals and use cases, and should meet psychological requirements and thus includes further psychological competencies in addition to the typical facets of AIL. We derived 60 items to represent different facets of AI Literacy according to Ng and colleagues conceptualisation of AI literacy and additional 12 items to represent psychological competencies such as problem solving, learning, and emotion regulation in regard to AI. For this purpose, data were collected online from 300 German-speaking adults. The items were tested for factorial structure in confirmatory factor analyses. The result is a measurement instrument that measures AI literacy with the facets Use & apply AI, Understand AI, Detect AI, and AI Ethics and the ability to Create AI as a separate construct, and AI Self-efficacy in learning and problem solving and AI Self-management. This study contributes to the research on AI literacy by providing a measurement instrument relying on profound competency models. In addition, higher-order psychological competencies are included that are particularly important in the context of pervasive change through AI systems.

Motivation & Objective

• To ground an AI literacy questionnaire in established competency models and psychological factors.
• To create a modular instrument with facets that can be used independently in professional contexts.
• To extend AI literacy measurement by incorporating psychological change- and meta-competencies alongside standard literacy facets.
• To derive and validate a 60-item AI-literacy item set plus 12 psychological-competency items.
• To provide a usable measurement tool for assessing AI literacy and related psychological competencies in real-world settings.

Proposed method

• Item development based on Ng et al.'s AI literacy framework plus additional psychological competencies.
• Online data collection from 300 German-speaking adults.
• Confirmatory factor analyses to test factorial structure of the item set.
• Model testing to establish a measurement instrument with multiple facets and constructs.
• Inclusion of broader constructs such as AI self-efficacy in learning and problem solving and AI self-management.

Experimental results

Research questions

• RQ1 Does MAILS capture distinct facets of AI literacy as defined by established frameworks (Use & Apply AI, Understand AI, Detect AI, AI Ethics, Create AI)?
• RQ2 Are the proposed psychological competencies (e.g., problem solving, learning, emotion regulation in relation to AI) measurably distinct and reliable?
• RQ3 Does confirmatory factor analysis support the hypothesized factorial structure and modularity of MAILS?
• RQ4 Can MAILS be used flexibly in professional settings due to its modular design while maintaining valid measurement properties?

Key findings

• MAILS yields a factorial structure comprising Use & Apply AI, Understand AI, Detect AI, AI Ethics, and Create AI as separate constructs.
• Psychological competencies related to AI (e.g., problem solving, learning, emotion regulation) are measurable alongside AI-literacy facets.
• The instrument demonstrates a validated measurement model in a sample of 300 German-speaking adults.
• The questionnaire is modular, enabling independent use of different facets for diverse goals and use cases in professional contexts.