[Paper Review] Measurement in AI Policy: Opportunities and Challenges

Saurabh Mishra, Jack Clark+1 more|arXiv (Cornell University)|Sep 10, 2020

Ethics and Social Impacts of AI33 references42 citations

TL;DR

This paper surveys measurement problems and opportunities in AI policy, based on a 2019 Stanford workshop, and identifies six core challenges to guide future research.

ABSTRACT

As artificial intelligence increasingly influences our world, it becomes crucial to assess its technical progress and societal impact. This paper surveys problems and opportunities in the measurement of AI systems and their impact, based on a workshop held at Stanford University in the fall of 2019. We identify six summary challenges inherent to measuring the progress and impact of AI, and summarize over 40 presentations and associated discussions from the workshop. We hope this can inspire research agendas in this crucial area.

Motivation & Objective

• Clarify how to measure AI progress and impact across technical, economic, and societal dimensions.
• Identify definitional and methodological bottlenecks that hinder robust AI measurement.
• Propose directions for improving data sources, metrics, and evaluation practices for policy use.
• Highlight how measurement informs governance, investment, and ethical considerations in AI.
• Encourage interdisciplinary collaboration to develop standardizable measurement frameworks.

Proposed method

• Synthesize insights from 40+ workshop presentations and breakout sessions.
• Organize findings around six core measurement challenges.
• Summarize plenary talks and subgroup discussions to extract common themes.
• Discuss practical measurement questions and potential research directions for each area.

Experimental results

Research questions

• RQ1 What constitutes AI and how should it be defined for measurement across sectors?
• RQ2 How can we meaningfully measure AI progress beyond single-metric evaluations?
• RQ3 What data and bibliometric approaches best capture AI impact and development trajectories?
• RQ4 How can we quantify AI’s economic and societal effects, including labor markets and inequality?
• RQ5 What metrics are needed to assess risks, governance, and human rights implications of AI?

Key findings

• AI progress is difficult to measure due to definitional ambiguity and reliance on dataset-specific benchmarks.
• Overfitting and bias on benchmarks can misrepresent real-world AI capabilities and safety concerns.
• Diverse metrics and adversarial evaluation are needed to capture robust progress beyond single scores.
• Bibliometric data can illuminate contributions, collaborations, and diffusion but require careful handling of authorship and geography.
• Economic and societal measurements must account for intangible inputs (skills, data, practices) and service-based outputs.
• Risks, governance, and human rights considerations require structured data and transparent methodologies for policy-relevant insights.