[Paper Review] Measuring Linguistic Diversity During COVID-19

Jonathan Dunn, Tom Coupé+1 more|UvA-DARE (University of Amsterdam)|Apr 3, 2021

Linguistics, Language Diversity, and Identity19 references20 citations

TL;DR

This paper proposes a difference-in-differences method using the Herfindahl-Hirschman Index (HHI) to calibrate linguistic diversity measures in digital corpora by exploiting travel restrictions during the COVID-19 pandemic as a natural experiment. It demonstrates that non-local populations—especially tourists—significantly distort linguistic diversity in social media data, and that 79.3% of observed changes in linguistic diversity during the pandemic are attributable to these restrictions rather than pre-existing macro-trends.

ABSTRACT

Computational measures of linguistic diversity help us understand the linguistic landscape using digital language data. The contribution of this paper is to calibrate measures of linguistic diversity using restrictions on international travel resulting from the COVID-19 pandemic. Previous work has mapped the distribution of languages using geo-referenced social media and web data. The goal, however, has been to describe these corpora themselves rather than to make inferences about underlying populations. This paper shows that a difference-in-differences method based on the Herfindahl-Hirschman Index can identify the bias in digital corpora that is introduced by non-local populations. These methods tell us where significant changes have taken place and whether this leads to increased or decreased diversity. This is an important step in aligning digital corpora like social media with the real-world populations that have produced them.

Motivation & Objective

• To address biases in digital language corpora—particularly non-local bias—by leveraging the pandemic as a natural experiment.
• To validate computational measures of linguistic diversity using real-world population shifts during travel restrictions.
• To identify which countries and languages are most affected by non-local populations in social media data.
• To distinguish pandemic-induced changes in linguistic diversity from pre-existing macro-trends such as immigration or bilingualism shifts.
• To provide a methodological framework for correcting non-local bias in digital corpora used for population and language mapping.

Proposed method

• Applies a difference-in-differences (DiD) approach comparing linguistic diversity in 2018, 2019, and 2020 to isolate pandemic-specific effects.
• Uses the Herfindahl-Hirschman Index (HHI) as a measure of linguistic diversity, where lower HHI indicates higher diversity.
• Analyzes geo-referenced Twitter data from a two-year period (2018–2020) to track changes in language distribution across countries.
• Establishes a seasonally-adjusted baseline using 2018 data to control for temporal fluctuations and seasonal variation.
• Identifies countries with significant changes in linguistic diversity during the pandemic (July–September 2020) and compares them to pre-pandemic trends.
• Corrects for production and sampling bias by comparing data volume and economic indicators (e.g., GDP, internet access) to population size and wealth.

Figure 1: Number of observations per country.

Experimental results

Research questions

• RQ1 To what extent do travel restrictions during the COVID-19 pandemic reduce non-local population influence in digital language corpora?
• RQ2 How can changes in linguistic diversity during the pandemic be distinguished from pre-existing macro-trends such as immigration or language shift?
• RQ3 Which countries and languages show the most significant changes in linguistic diversity due to the departure of non-local populations?
• RQ4 Can the Herfindahl-Hirschman Index (HHI) be reliably used to measure and correct for non-local bias in social media data?
• RQ5 To what extent does the observed shift in linguistic diversity reflect changes in population composition rather than changes in language use behavior?

Key findings

• 79.3% of the 58 countries showing significant changes in linguistic diversity during the pandemic (July–September 2020) had no such change in the 2018–2019 baseline period, indicating the changes are primarily due to pandemic-related travel restrictions.
• In New Zealand and Australia, the proportion of English in tweets decreased from 86.26% to 84.13% and 89.51% to 87.45%, respectively, while non-local English-speaking tourists left, allowing minority languages like Spanish and Portuguese to become more prominent.
• The study identifies that non-local bias—especially from tourists—significantly distorts linguistic diversity in digital corpora, particularly in countries with high tourist inflows like New Zealand and Australia.
• The HHI-based measure of linguistic diversity successfully captures real-world population shifts, validating its use in correcting for non-local bias in digital corpora.
• Production and sampling biases were controlled for using population and economic data, confirming that the observed changes are not due to data volume imbalances or wealth-based sampling effects.
• The method successfully isolates pandemic-specific changes from broader demographic trends, demonstrating that digital corpora can be calibrated to reflect real-world linguistic diversity when using natural experiments like travel restrictions.

Figure 2: Geographic distribution of data by region by month.