[Paper Review] Human-Centric Research for NLP: Towards a Definition and Guiding Questions

Bhushan Kotnis, Kiril Gashteovski+8 more|arXiv (Cornell University)|Jul 10, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a working definition of Human-Centric Research (HCR) in NLP, emphasizing active involvement of external stakeholders—such as end users, domain experts, and data collectors—throughout the research pipeline. It outlines four research stages—problem definition, data collection, model building, and evaluation—each with guiding questions to embed human needs, and demonstrates their application in document similarity and information extraction tasks.

ABSTRACT

With Human-Centric Research (HCR) we can steer research activities so that the research outcome is beneficial for human stakeholders, such as end users. But what exactly makes research human-centric? We address this question by providing a working definition and define how a research pipeline can be split into different stages in which human-centric components can be added. Additionally, we discuss existing NLP with HCR components and define a series of guiding questions, which can serve as starting points for researchers interested in exploring human-centric research approaches. We hope that this work would inspire researchers to refine the proposed definition and to pose other questions that might be meaningful for achieving HCR.

Motivation & Objective

• To address the lack of a clear, actionable definition of Human-Centric Research (HCR) in NLP.
• To identify and structure key stages in the NLP research pipeline where external stakeholders can be meaningfully involved.
• To provide a set of guiding research questions that help researchers design HCR projects with real-world impact.
• To demonstrate the practical application of HCR principles through case studies in information extraction and document similarity.
• To inspire the NLP community to refine HCR definitions and explore new, stakeholder-driven research directions.

Proposed method

• Proposes a working definition: HCR occurs when external stakeholders actively participate in research, beyond researchers alone.
• Splits the NLP research pipeline into four stages—problem definition, data collection, model building, and evaluation—each with potential HCR integration points.
• Introduces a series of structured, domain-agnostic guiding questions for each stage to help researchers embed stakeholder needs.
• Adapts the guiding questions into task-specific examples for information extraction and semantic textual similarity (STS) tasks.
• Uses case studies to illustrate how HCR principles can be operationalized in real NLP applications.
• Emphasizes iterative, participatory design by involving stakeholders in defining problems, labeling data, shaping model behavior, and evaluating outcomes.

Experimental results

Research questions

• RQ1 What defines a research project as human-centric in NLP, and how can this be operationalized across the research pipeline?
• RQ2 How can external stakeholders be meaningfully involved in each stage of NLP research—problem definition, data collection, model building, and evaluation?
• RQ3 What specific guiding questions can help researchers shift from intuition-driven to stakeholder-driven NLP research?
• RQ4 How can HCR principles be adapted to specific NLP tasks such as information extraction and document similarity?
• RQ5 In what ways can HCR improve the practical relevance and real-world impact of NLP systems?

Key findings

• The paper establishes a working definition of HCR as research where external stakeholders actively participate, not just as passive subjects but as co-creators.
• The four-stage research pipeline (problem definition, data collection, model building, evaluation) provides a structured framework for integrating HCR components.
• Guiding questions for each stage help researchers identify stakeholder needs and align research goals with real-world user pain points.
• Case studies in information extraction and document similarity show that HCR questions can be concretely applied to improve model design and evaluation criteria.
• The approach enables researchers to communicate the human-centric nature of their work more clearly and to ensure that outcomes are relevant and beneficial to end users.
• The framework is intentionally flexible and open-ended, encouraging community refinement and adaptation to diverse NLP applications.