[Paper Review] Generative AI, Pragmatics, and Authenticity in Second Language Learning

Robert Godwin‐Jones|arXiv (Cornell University)|Oct 18, 2024

Language, Metaphor, and Cognition4 citations

TL;DR

This paper examines the limitations of generative AI in second language learning, particularly its inability to produce pragmatically appropriate and culturally authentic language due to statistical modeling and biased training data. It argues that while AI offers benefits in tutoring and material creation, its lack of lived social experience and cultural awareness undermines its effectiveness in developing intercultural communication competence.

ABSTRACT

There are obvious benefits to integrating generative AI (artificial intelligence) into language learning and teaching. Those include using AI as a language tutor, creating learning materials, or assessing learner output. However, due to how AI systems under-stand human language, based on a mathematical model using statistical probability, they lack the lived experience to be able to use language with the same social aware-ness as humans. Additionally, there are built-in linguistic and cultural biases based on their training data which is mostly in English and predominantly from Western sources. Those facts limit AI suitability for some language learning interactions. Stud-ies have clearly shown that systems such as ChatGPT often do not produce language that is pragmatically appropriate. The lack of linguistic and cultural authenticity has important implications for how AI is integrated into second language acquisition as well as in instruction targeting development of intercultural communication compe-tence.

Motivation & Objective

• To investigate how generative AI systems perform in pragmatic language use within second language learning contexts.
• To analyze the impact of training data biases—especially English and Western-centric sources—on AI-generated language output.
• To evaluate the suitability of generative AI for developing intercultural communication competence in language learners.
• To identify gaps in AI's ability to replicate authentic, socially aware language use compared to native speakers.
• To guide educators in responsibly integrating AI tools while acknowledging their limitations in fostering cultural and pragmatic authenticity.

Proposed method

• The paper conducts a critical analysis of existing studies on generative AI performance in language tasks, focusing on pragmatic appropriateness.
• It evaluates AI outputs using criteria for sociolinguistic and cultural authenticity, particularly in context-sensitive language use.
• The analysis draws on linguistic theory and second language acquisition (SLA) research to assess AI's alignment with native-like pragmatic competence.
• The study examines training data composition, emphasizing dominance of English and Western cultural content in large language models.
• It compares AI-generated responses with native speaker norms to identify discrepancies in tone, formality, and cultural relevance.
• The methodology relies on qualitative and comparative evaluation rather than experimental data collection.

Experimental results

Research questions

• RQ1 To what extent do generative AI systems produce pragmatically appropriate language in second language learning contexts?
• RQ2 How do linguistic and cultural biases in AI training data affect the authenticity of generated language?
• RQ3 Can generative AI effectively support the development of intercultural communication competence in language learners?
• RQ4 What are the limitations of AI in replicating the social awareness and lived experience inherent in authentic language use?
• RQ5 How should educators integrate generative AI in language teaching given its shortcomings in pragmatics and authenticity?

Key findings

• Generative AI systems such as ChatGPT frequently produce language that is pragmatically inappropriate, despite grammatical correctness.
• The training data for most large language models is predominantly in English and sourced from Western cultural contexts, introducing systemic linguistic and cultural biases.
• AI lacks lived social experience, which limits its ability to use language with the same level of social awareness as human speakers.
• These limitations significantly reduce AI's effectiveness in teaching authentic, context-sensitive language use essential for intercultural communication.
• The absence of cultural and pragmatic authenticity in AI outputs undermines its utility in advanced second language acquisition beyond basic language practice.
• Educators must critically evaluate AI-generated content for authenticity and pragmatics when integrating it into language instruction.