[Paper Review] Conversational AI Threads for Visualizing Multidimensional Datasets

Matt-Heun Hong, Anamaria Crisan|arXiv (Cornell University)|Nov 9, 2023

AI in Service Interactions4 citations

TL;DR

This paper introduces AI Threads, a multi-threaded conversational AI system that enhances visual data analysis by enabling users to manage multiple analytic conversations simultaneously. Using large language models (LLMs), AI Threads supports dynamic, context-aware refinement of visualizations, demonstrating improved usability and coherence in complex analytic dialogues, though challenges like hallucinations and context drift remain.

ABSTRACT

Generative Large Language Models (LLMs) show potential in data analysis, yet their full capabilities remain uncharted. Our work explores the capabilities of LLMs for creating and refining visualizations via conversational interfaces. We used an LLM to conduct a re-analysis of a prior Wizard-of-Oz study examining the use of chatbots for conducting visual analysis. We surfaced the strengths and weaknesses of LLM-driven analytic chatbots, finding that they fell short in supporting progressive visualization refinements. From these findings, we developed AI Threads, a multi-threaded analytic chatbot that enables analysts to proactively manage conversational context and improve the efficacy of its outputs. We evaluate its usability through a crowdsourced study (n=40) and in-depth interviews with expert analysts (n=10). We further demonstrate the capabilities of AI Threads on a dataset outside the LLM's training corpus. Our findings show the potential of LLMs while also surfacing challenges and fruitful avenues for future research.

Motivation & Objective

• To investigate how LLMs handle shifting conversational context during visual data analysis.
• To address the limitations of single-threaded chatbots in managing progressive visualization refinements.
• To design and evaluate a multi-threaded conversational interface that improves context management and analytic coherence.
• To assess the usability and reliability of LLM-powered chatbots in real-world data analysis tasks.
• To identify key challenges and future directions for AI-driven visual analytic tools.

Proposed method

• Re-analyzed a prior Wizard-of-Oz study to extract conversational context shifts and ground truth for LLM behavior evaluation.
• Designed AI Threads as a multi-threaded chatbot interface where users can create separate threads for distinct analytic tasks.
• Integrated an LLM as the AI agent to generate both textual responses and visualization code in response to natural language queries.
• Implemented thread-specific context management, allowing users to isolate and refine different aspects of an analysis.
• Used a combination of crowdsourced studies (n=40) and expert interviews (n=10) to evaluate usability, accuracy, and user perception.
• Evaluated the system on the Titanic dataset (within LLM’s training distribution) and a new dataset (outside training data) to test generalization.

Figure 1. Excerpts of participants’ analytic conversations with AI Threads. We show participant conversations in threads modifying the visual encoding (E03, E04, P40), as well as for sequential (E07) and individual (P32) utterances. Excerpts were derived from participants in crowd-sourced (P#) and i

Experimental results

Research questions

• RQ1 How do LLMs perform in maintaining conversational context during progressive visualization refinements in data analysis?
• RQ2 What are the strengths and limitations of single-threaded LLM chatbots in supporting complex analytic dialogues?
• RQ3 To what extent can multi-threaded conversation management improve the usability and accuracy of AI-driven visual analytic tools?
• RQ4 How do users perceive and interact with an LLM-powered chatbot that supports both text and visualization generation?
• RQ5 What challenges—particularly hallucinations and context drift—emerge when using LLMs for data analysis, and how can they be mitigated?

Key findings

• AI Threads demonstrated strong capability in generating and refining visualizations through natural language, supporting operations like attribute addition/removal and encoding type changes.
• Participants in the crowdsourced study (n=40) reported a positive perception of AI Threads, with high usability and perceived usefulness in visual analysis tasks.
• Expert analysts (n=10) confirmed that AI Threads supports richer, more flexible analytic dialogues but highlighted the need for better control over dialogue flow and provenance.
• Despite overall accuracy, the system exhibited non-obvious errors, such as incorrect data modifications (e.g., P40) and misleading explanations (e.g., P35), indicating persistent hallucination risks.
• Performance was strong on the Titanic dataset (within training distribution), but generalization to out-of-distribution datasets remains a challenge.
• The multi-threaded design significantly improved context management compared to single-threaded alternatives, enabling users to organize complex analyses more effectively.

Figure 2. Comparison for visualization produced between the WoZ study and different systems. Progressive utterances for modifying an initial visual encoding (top row) by different systems. Black boxes around the visualization indicate the creation of a new visualization—a failure to apply a progress