[Paper Review] Frontiers of Information Access Experimentation for Research and Education (Dagstuhl Seminar 23031)

Christine Bauer, Ben Carterette+2 more|arXiv (Cornell University)|Jan 1, 2023

Semantic Web and OntologiesComputer Science24 citations

TL;DR

This paper proposes a comprehensive framework to improve experimental rigor and education in information access research by addressing methodological shortcomings in information retrieval, recommender systems, and NLP. It introduces five key initiatives—real-world evaluation, human-machine relevance judgment, methodological education, results-blind reviewing, and author guidance—supported by community-driven, living guidelines to enhance validity, fairness, and reproducibility in research.

ABSTRACT

This report documents the program and the outcomes of Dagstuhl Seminar 23031 ``Frontiers of Information Access Experimentation for Research and Education'', which brought together 37 participants from 12 countries. The seminar addressed technology-enhanced information access (information retrieval, recommender systems, natural language processing) and specifically focused on developing more responsible experimental practices leading to more valid results, both for research as well as for scientific education. The seminar brought together experts from various sub-fields of information access, namely IR, RS, NLP, information science, and human-computer interaction to create a joint understanding of the problems and challenges presented by next generation information access systems, from both the research and the experimentation point of views, to discuss existing solutions and impediments, and to propose next steps to be pursued in the area in order to improve not also our research methods and findings but also the education of the new generation of researchers and developers. The seminar featured a series of long and short talks delivered by participants, who helped in setting a common ground and in letting emerge topics of interest to be explored as the main output of the seminar. This led to the definition of five groups which investigated challenges, opportunities, and next steps in the following areas: reality check, i.e. conducting real-world studies, human-machine-collaborative relevance judgment frameworks, overcoming methodological challenges in information retrieval and recommender systems through awareness and education, results-blind reviewing, and guidance for authors.

Motivation & Objective

• . The research addresses the growing need for more responsible, valid, and educationally sound experimental practices in information access research.
• It seeks to improve internal, construct, and external validity in experimental evaluation of IR, RS, and NLP systems.
• The objective includes fostering collaboration between academia and industry in experimentation and evaluation.
• It aims to develop educational resources to equip the next generation of researchers with critical experimentation skills.
• The paper proposes living, community-maintained guidelines to standardize and improve the quality of research submissions and reviews in the field.

Proposed method

• . The seminar convened 37 experts from 12 countries to identify core challenges in experimental practices across IR, RS, and NLP.
• Five working groups were formed to investigate: real-world studies, human-machine relevance judgment, methodological education, results-blind reviewing, and author guidance.
• The authors developed a draft set of concise, broad, and constructive guidelines for authors, emphasizing motivation, methodological justification, and reproducibility.
• The guidelines emphasize clear claims, literature review depth, methodological reasoning, and ethical data use.
• Results-blind reviewing is proposed to shift focus from performance gains to theoretical soundness, hypothesis quality, and methodological design.
• The guidelines are intended as living documents, open to revision and community consultation, with recommendations for integration into conference and journal review processes.

Experimental results

Research questions

• RQ1 . What are the most promising experimentation methodologies to advance responsible research in information access?
• RQ2 How can fairness, accountability, and transparency (FAccT) be embedded into experimental practices (FAccT-E)?
• RQ3 What are effective models for collaboration between academia and industry in experimentation?
• RQ4 How can critical experimentation skills be systematically taught in academic education?
• RQ5 How can shared evaluation infrastructures and hybrid participation models be designed to support collaborative research?

Key findings

• . The proposed author guidance emphasizes clear motivation, well-scoped claims, and explicit distinction from prior work, with a focus on methodological justification over performance gains.
• . The guidelines recommend using diverse, publicly available datasets and providing sufficient detail on data sources, preparation, and code to ensure reproducibility.
• . Results-blind reviewing is proposed to reduce bias toward performance-driven outcomes and to prioritize strong hypotheses, methodological design, and analysis plans.
• . The working group identified that real-world studies face challenges in recruitment, data representation, and longitudinal design, requiring new infrastructure and domain-specific approaches.
• . Human-machine collaboration in relevance judgment shows promise, but conditions under which LLMs can replace human assessors remain unclear and require further research.
• . The community is encouraged to treat the guidelines as living documents, with regular updates and broad consultation to reflect evolving research norms and ethical standards.