[Paper Review] Measuring Diversity of Artificial Intelligence Conferences

Ana Freire, Lorenzo Porcaro+1 more|arXiv (Cornell University)|Jan 20, 2020

Ethics and Social Impacts of AI12 references20 citations

TL;DR

This paper proposes a set of diversity indicators—gender, geographical, and business diversity—for measuring and monitoring diversity in major AI conferences. Using data from authors, keynote speakers, and organizers, it computes a Conference Diversity Index (CDI) that reveals significant underrepresentation of African researchers and gender imbalance, while showing progress in keynote diversity, with NeurIPS 2020 achieving the highest CDI (0.89).

ABSTRACT

The lack of diversity of the Artificial Intelligence (AI) field is nowadays a concern, and several initiatives such as funding schemes and mentoring programs have been designed to overcome it. However, there is no indication on how these initiatives actually impact AI diversity in the short and long term. This work studies the concept of diversity in this particular context and proposes a small set of diversity indicators (i.e. indexes) of AI scientific events. These indicators are designed to quantify the diversity of the AI field and monitor its evolution. We consider diversity in terms of gender, geographical location and business (understood as the presence of academia versus industry). We compute these indicators for the different communities of a conference: authors, keynote speakers and organizing committee. From these components we compute a summarized diversity indicator for each AI event. We evaluate the proposed indexes for a set of recent major AI conferences and we discuss their values and limitations.

Motivation & Objective

• To address the lack of standardized metrics for measuring and monitoring diversity in AI scientific communities.
• To quantify diversity in major AI conferences across three dimensions: gender, geographical origin, and business sector (academia vs. industry).
• To develop a composite Conference Diversity Index (CDI) that summarizes diversity across multiple components (authors, speakers, organizers).
• To evaluate the proposed indicators on real-world data from top AI conferences and assess their utility for policy and organizational action.
• To enable long-term tracking of diversity trends and support targeted interventions to improve representation.

Proposed method

• Define three core diversity indicators: Gender Diversity Index (GDI), Geographical Diversity Index (GeoDI), and Business Diversity Index (BDI), each based on category distribution.
• Use the Shannon entropy formula to compute each index, with normalization to ensure comparability across conferences.
• Collect data on authors, keynote speakers, and organizing committees from 12 major AI conferences (e.g., NeurIPS, ICML, RecSys) between 2018 and 2020.
• Aggregate individual component indices (authors, speakers, organizers) into a single Conference Diversity Index (CDI) using a weighted average.
• Apply gender proxies based on name-based classification (male, female, non-binary), acknowledging potential errors.
• Use country affiliations as a proxy for geographical origin, grouping into regions (e.g., North America, Europe, Asia, Africa, Latin America, Oceania).

Experimental results

Research questions

• RQ1 How can diversity in AI conferences be systematically measured across gender, geography, and business sector?
• RQ2 To what extent do major AI conferences exhibit underrepresentation of specific groups, particularly in Africa and among women?
• RQ3 Can a composite diversity index effectively summarize and compare diversity across different conferences?
• RQ4 How has keynote speaker diversity evolved in recent years, especially in terms of gender balance?
• RQ5 What are the limitations of current proxies (e.g., name-based gender, affiliation-based geography) in measuring diversity?

Key findings

• NeurIPS 2020 achieved the highest Conference Diversity Index (CDI) of 0.89, indicating strong diversity across all dimensions.
• The lowest CDI was recorded for ICML 2018 (0.44) and RecSys 2019 (0.49), primarily due to low business diversity with all keynote speakers from academia.
• Only 5 researchers from Africa were found among authors and organizers across all conferences, highlighting severe underrepresentation.
• Keynote speaker diversity improved significantly, with several conferences achieving near-gender balance, suggesting progress in visibility efforts.
• Geographical diversity remains low, with most conferences dominated by researchers from North America and Europe, and only 3–4 regions represented on average.
• The Conference Diversity Index (CDI) provides a concise, comparable metric that enables tracking diversity trends and identifying areas for improvement.