[Paper Review] Measure of gap and inequalities in basic education students proficiencies

José Francisco Soares, Érica Castilho Rodrigues+1 more|arXiv (Cornell University)|May 24, 2018

Intergenerational and Educational Inequality Studies13 references3 citations

TL;DR

This paper proposes a novel methodology using Kullback-Leibler divergence to simultaneously measure learning quality and socioeconomic inequality in basic education by comparing student proficiency distributions against a reference distribution (OECD PISA performance). The approach identifies municipalities where low performance or high inequality—often overlooked—drives educational exclusion, revealing that quality and inequality must be addressed together in policy.

ABSTRACT

This study uses students performance on standardized tests as evidence of the quality of education and introduces a methodology based on the comparison of performance distributions to produce indicators for both the level achieved by the students and the learning gap between social groups, two inseparable dimensions of quality of education. In the first case, the study compares the distribution of the group observed with a reference distribution, which represents an ideal situation of where students should be. In the second, it compares the performance distribution of students belonging to social groups defined by socioeconomic characteristics. This article uses the Kullback-Leibler divergence to characterize the differences between the distributions. This measure takes into account types of diferences not considered by other measures and have solid conceptual justifications. The proposed methodology is used to describe the quality of Brazilian basic education using the test results applied biannually to all Brazilian students of basic education.

Motivation & Objective

• To develop a unified framework for measuring both learning quality and socioeconomic inequality in education, moving beyond isolated indicators.
• To address the limitation of traditional inequality measures (e.g., Theil index) that assume redistributability and uniformity, which do not apply in educational contexts.
• To define a context-specific reference distribution of proficiency levels based on international benchmarks (OECD PISA) rather than arbitrary ideals.
• To identify municipalities where educational exclusion stems primarily from low average performance versus high inequality between social groups.
• To provide policy-relevant indicators that distinguish between quality and inequality problems, enabling targeted interventions.

Proposed method

• Uses Kullback-Leibler (KL) divergence to quantify the distance between empirical student proficiency distributions and a reference distribution representing ideal learning outcomes.
• Defines the reference distribution as the performance distribution of a typical OECD country from PISA assessments, reflecting curricular expectations.
• Applies KL divergence to measure two distinct distances: (1) between the actual student distribution and the reference (quality gap), and (2) between high- and low-SES student distributions (inequality gap).
• Treats proficiency as a continuous performance metric rather than individual-level outcomes, avoiding assumptions of redistributability or fixed total knowledge.
• Uses biannual national test data from Brazil’s basic education system to compute these indicators at the municipal level.
• Visualizes results via scatter plots (e.g., SES gap vs. KL divergence) to classify municipalities by dominant exclusion type—low performance or high inequality.

Experimental results

Research questions

• RQ1 To what extent do Brazilian municipalities exhibit a learning gap between students of different socioeconomic statuses, independent of overall performance levels?
• RQ2 How can educational quality and inequality be measured simultaneously using distributional comparisons rather than summary statistics?
• RQ3 In which municipalities is the primary driver of educational exclusion due to low average performance, and in which is it due to high inequality between social groups?
• RQ4 Can the Kullback-Leibler divergence effectively capture nuanced differences in proficiency distributions that traditional inequality measures miss?
• RQ5 What policy implications arise when quality and inequality are measured as distinct but interrelated dimensions of educational performance?

Key findings

• Most Brazilian municipalities exhibit both a significant learning gap (low average performance) and high inequality between high- and low-SES students, indicating widespread intraschool exclusion.
• Only a few municipalities—such as Teresina and Salvador—show low inequality, but even these have substantial performance gaps, indicating that low inequality does not imply high quality.
• The methodology successfully distinguishes between municipalities where the main problem is low average learning (e.g., Teresina) versus those where the main issue is high inequality (e.g., some capitals with high KL divergence despite moderate performance).
• The application of KL divergence captures distributional differences not detected by standard inequality indices, such as shifts in the shape or spread of proficiency distributions.
• The results show that improving overall learning levels through policy can simultaneously reduce inequality, challenging the assumption that solving quality automatically resolves inequality.
• The study reveals that current educational debates in Brazil often overlook inequality by focusing solely on average performance, leading to misaligned policy priorities.