[Paper Review] Large Language Models Reflect the Ideology of their Creators

Maarten Buyl, Alexander Rogiers+9 more|arXiv (Cornell University)|Oct 24, 2024

Computational and Text Analysis Methods10 citations

TL;DR

The paper analyzes ideological stances of 17 popular LLMs across English and Chinese, showing language prompts and creator region influence model ideologies, with substantial cross-model differences. It argues for transparency about design choices and cautions against assuming neutrality.

ABSTRACT

Large language models (LLMs) are trained on vast amounts of data to generate natural language, enabling them to perform tasks like text summarization and question answering. These models have become popular in artificial intelligence (AI) assistants like ChatGPT and already play an influential role in how humans access information. However, the behavior of LLMs varies depending on their design, training, and use. In this paper, we prompt a diverse panel of popular LLMs to describe a large number of prominent personalities with political relevance, in all six official languages of the United Nations. By identifying and analyzing moral assessments reflected in their responses, we find normative differences between LLMs from different geopolitical regions, as well as between the responses of the same LLM when prompted in different languages. Among only models in the United States, we find that popularly hypothesized disparities in political views are reflected in significant normative differences related to progressive values. Among Chinese models, we characterize a division between internationally- and domestically-focused models. Our results show that the ideological stance of an LLM appears to reflect the worldview of its creators. This poses the risk of political instrumentalization and raises concerns around technological and regulatory efforts with the stated aim of making LLMs ideologically 'unbiased'.

Motivation & Objective

• Investigate whether LLMs reflect the ideology of their creators across languages and regions.
• Quantify moral assessments of controversial historical figures produced by diverse LLMs.
• Examine the influence of prompting language (English vs Chinese) on LLM ideological positions.
• Assess differences between Western and non-Western models in terms of ideology.
• Discuss implications for regulation, transparency, and model development.

Proposed method

• Two-stage open-ended elicitation: Stage 1 has LLM describe a political person; Stage 2 asks the LLM to rate any moral assessment in the Stage 1 text.
• Panel of 17 LLMs evaluated in English and Chinese (listed in Table 2).
• Political persons selected from Pantheon dataset (4,339 figures) with multi-criteria filtering and popularity thresholds.
• Annotations with Manifesto Project tags (61 tags) to aid interpretability of political orientations.
• Data quality checks linking Stage 1 descriptions to Wikipedia summaries and ensuring Stage 2 follows Likert-scale prompts.

Figure 2 : Biplot showing the two-dimensional PCA-projection of the respondent’s average score for each ideology tag, with the factor loadings visualized as a grey vector that has a thickness proportional to the loading’s norm. To clarify the effect of the prompting language, Chinese respondents are

Experimental results

Research questions

• RQ1 Do LLMs exhibit systematic ideological differences across languages (English vs Chinese) when describing political figures?
• RQ2 Do Western vs non-Western LLMs differ in their evaluations of political persons and alignment with liberal democratic values?
• RQ3 How do prompting language and model origin interact to shape attitudes toward specific ideologies and political actors?
• RQ4 Can we quantify and visualize ideological diversity across a broad set of LLMs using open-ended elicitation?

Key findings

• Chinese prompting generally yields more favorable views toward China-aligned figures and centralized governance traits.
• Western models tend to rate liberal democratic values and human rights-related tags more positively than non-Western models when prompted in English.
• Within Western models, notable ideological variation exists between OpenAI, Gemini, Mistral, and Anthropic families, with different emphases on inclusivity, governance, and corruption.
• Prompting language explains a significant portion of variance in LLM ideology (p = 0.0008 for Chinese vs English differences).
• Non-Western models show relatively more support for centralized economic governance and national stability, while Western models favor individual liberties and social justice.
• There is evidence of cross-language (English vs Chinese) and cross-region (Western vs non-Western) ideological alignment across models.

Figure 3 : Average score difference over all respondents prompted in Chinese versus English. Red line indicates overall mean difference. Only the top 20 most positive and top 20 most negative differences are shown.