[Paper Review] How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study

Ken Gu, Madeleine Grunde-McLaughlin+3 more|arXiv (Cornell University)|Sep 18, 2023

Mobile Crowdsensing and CrowdsourcingComputer Science3 citations

TL;DR

This study investigates how data analysts respond to AI-assisted planning during data analysis through a Wizard-of-Oz experiment with 13 experienced analysts. It finds that contextual, goal-aligned planning suggestions—particularly those prompting reflection on alternative models or rationales—significantly enhance analytical robustness, but effectiveness depends heavily on timing, framing, and trust-building mechanisms, highlighting key design principles for future AI assistants in data science workflows.

ABSTRACT

Data analysis is challenging as analysts must navigate nuanced decisions that may yield divergent conclusions. AI assistants have the potential to support analysts in planning their analyses, enabling more robust decision making. Though AI-based assistants that target code execution (e.g., Github Copilot) have received significant attention, limited research addresses assistance for both analysis execution and planning. In this work, we characterize helpful planning suggestions and their impacts on analysts' workflows. We first review the analysis planning literature and crowd-sourced analysis studies to categorize suggestion content. We then conduct a Wizard-of-Oz study (n=13) to observe analysts' preferences and reactions to planning assistance in a realistic scenario. Our findings highlight subtleties in contextual factors that impact suggestion helpfulness, emphasizing design implications for supporting different abstractions of assistance, forms of initiative, increased engagement, and alignment of goals between analysts and assistants.

Motivation & Objective

• Understand how data analysts respond to AI-generated planning suggestions during real-world data analysis tasks.
• Identify which types of planning suggestions are perceived as most helpful and under what contextual conditions.
• Investigate the role of assistant initiative, goal alignment, and engagement in shaping analysts’ trust and adoption of AI assistance.
• Address the gap in AI support for analysis planning—beyond code execution—by characterizing actionable suggestion content and design principles.
• Explore the feasibility and impact of integrating planning assistance into existing data analysis workflows to improve robustness and reduce arbitrary decision-making.

Proposed method

• Conducted a Wizard-of-Oz study with 13 experienced data analysts using a realistic, in-lab setting to simulate AI-assisted analysis on a real dataset.
• Designed a custom assistant interface that generated context-aware planning suggestions based on literature-derived categories of suggestion content.
• Used a semi-structured task involving data wrangling, modeling, and inference to elicit naturalistic analyst behavior and feedback.
• Collected qualitative and observational data on suggestion reception, trust, and workflow changes, with analysts interacting with a simulated AI assistant.
• Employed a behavior-driven approach to categorize suggestion types based on analysis planning literature and crowd-sourced studies.
• Implemented an open-source assistant interface to support reproducibility and future integration of planning assistance in data science tools.

Figure 1. Execution and planning assistance during data analysis . During an ongoing analysis (current step shown by the analyst icon), analysts may be focusing on the execution of an analysis decision (left) or planning their next decisions (right). During execution, analysts have a clear intent of

Experimental results

Research questions

• RQ1 Which categories of planning suggestions are perceived as most helpful by data analysts during real analysis workflows?
• RQ2 How do contextual factors—such as timing, framing, and goal alignment—affect analysts’ receptiveness to AI-generated planning suggestions?
• RQ3 What role does assistant initiative play in shaping analysts’ engagement and trust in AI-assisted planning?
• RQ4 How do analysts balance reliance on AI suggestions with their own analytical judgment and domain expertise?
• RQ5 In what ways can AI assistance support analysts in identifying and evaluating alternative analytical approaches to improve robustness?

Key findings

• Analysts found planning suggestions that prompted reflection on alternative models, rationales, or assumptions to be highly valuable, especially when they revealed overlooked analytical trade-offs.
• Suggestion helpfulness was strongly influenced by contextual factors such as timing, framing, and alignment with the analyst’s current analytical goal, with poorly timed or off-target suggestions reducing trust.
• Analysts were more likely to engage with suggestions when they perceived the assistant as goal-aligned and when suggestions encouraged deeper reasoning rather than just code generation.
• Despite high proficiency in statistics and programming, analysts often failed to recognize or consider the decision points highlighted by the assistant, indicating a need for more intuitive and proactive planning support.
• The study revealed a risk of overreliance on AI suggestions when they were framed as authoritative, underscoring the need for design mechanisms that promote critical evaluation and transparency.
• The most effective suggestions were those that scaffolded reasoning—such as prompting analysts to consider multiple modeling approaches—rather than simply recommending actions or code.

Figure 2. Planning assistance has limited support and is unexplored . Existing assistants, such as Github Copilot, are well-suited for execution assistance, such as suggesting data wrangling or modeling code.