[Paper Review] Digital Collaborator: Augmenting Task Abstraction in Visualization Design with Artificial Intelligence

Aditeya Pandey, Yixuan Zhang+3 more|arXiv (Cornell University)|Mar 2, 2020

Data Visualization and Analytics4 citations

TL;DR

This paper proposes a conceptual AI-powered Digital Collaborator (DC) to assist visualization researchers in automating and validating task abstraction—translating domain-specific goals into generalized, theory-based visualization tasks. Using natural language processing and task-abstraction frameworks, the DC provides rationale-backed recommendations to reduce human bias and improve consistency in design studies.

ABSTRACT

In the task abstraction phase of the visualization design process, including in "design studies", a practitioner maps the observed domain goals to generalizable abstract tasks using visualization theory in order to better understand and address the users needs. We argue that this manual task abstraction process is prone to errors due to designer biases and a lack of domain background and knowledge. Under these circumstances, a collaborator can help validate and provide sanity checks to visualization practitioners during this important task abstraction stage. However, having a human collaborator is not always feasible and may be subject to the same biases and pitfalls. In this paper, we first describe the challenges associated with task abstraction. We then propose a conceptual Digital Collaborator: an artificial intelligence system that aims to help visualization practitioners by augmenting their ability to validate and reason about the output of task abstraction. We also discuss several practical design challenges of designing and implementing such systems

Motivation & Objective

• Address the challenge of subjective, error-prone manual task abstraction in visualization design, which is vulnerable to designer bias and knowledge gaps.
• Overcome limitations of human collaborators, who may lack up-to-date knowledge or be subject to the same cognitive biases.
• Develop an AI-driven system that supports visualization practitioners in selecting appropriate task-abstraction frameworks and validating their abstractions.
• Enhance transparency and trust in AI recommendations through explainable outputs and confidence scoring.
• Explore equity considerations in AI-assisted task abstraction to ensure inclusivity of diverse domain experiences and perspectives.

Proposed method

• Design a question-and-answer interface modeled after Intelligent Personal Assistants (IPAs), where practitioners input domain-specific goals from interviews or observations.
• Employ natural language processing (NLP) to map input domain goals to generalized task descriptions using established task-abstraction frameworks such as the Multi-Level Typology, Graph Task Taxonomy, and Hierarchical Task Abstraction (HTA).
• Implement a framework characterization module to select the most appropriate abstraction framework based on input context, data type, and task characteristics.
• Generate multiple alternative abstractions with rationales to improve transparency and support user validation.
• Integrate confidence scores and human-in-the-loop mechanisms to address disagreement and improve trust in AI recommendations.
• Leverage web-crawling tools to extract training data from existing literature, with human oversight to ensure data quality and resolve conflicting abstractions.

Experimental results

Research questions

• RQ1 How can an AI system effectively recommend the most appropriate task-abstraction framework for a given domain goal?
• RQ2 What mechanisms can improve transparency and user trust in AI-generated task abstractions?
• RQ3 How can AI-based systems mitigate biases in task abstraction while maintaining alignment with visualization theory?
• RQ4 What role can human-in-the-loop validation play in refining and improving AI-generated abstractions?
• RQ5 How can equity and inclusivity be embedded in AI-driven task abstraction to reflect diverse domain experiences and design requirements?

Key findings

• The proposed Digital Collaborator can assist in mapping domain-specific goals to generalized visualization tasks using established abstraction frameworks, reducing reliance on manual, error-prone processes.
• Providing multiple alternative abstractions with rationales increases transparency and supports user validation, helping practitioners assess the reasonableness of AI suggestions.
• Confidence scores on recommendations can enhance user trust and reduce rejection of AI-generated outputs.
• Training data can be collected via smart web crawlers from research literature, though data quality issues such as conflicting abstractions require human oversight.
• Equity concerns in AI, including biased datasets and lack of representation, must be proactively addressed to ensure the system supports diverse design needs and perspectives.
• The system does not replace human-centered design practices but augments them by supporting the task abstraction phase with up-to-date, theory-grounded recommendations.