[Paper Review] Vi(E)va LLM! A Conceptual Stack for Evaluating and Interpreting Generative AI-based Visualizations

Luca Podo, Muhammad Ishmal+1 more|arXiv (Cornell University)|Feb 3, 2024

Scientific Computing and Data ManagementDecision Sciences3 citations

TL;DR

This paper introduces EvaLLM, a conceptual evaluation stack that decomposes the assessment of LLM-generated visualizations into atomic, interpretable components, enabling fine-grained, multi-faceted evaluation. The framework is implemented in a web-based platform supporting automated and manual scoring, demonstrating its utility through two case studies on GPT-3.5-turbo and Llama2-70b, which reveal structural and semantic errors in generated visualizations, thus establishing a foundation for standardized benchmarking in LLM-based visualization.

ABSTRACT

The automatic generation of visualizations is an old task that, through the years, has shown more and more interest from the research and practitioner communities. Recently, large language models (LLM) have become an interesting option for supporting generative tasks related to visualization, demonstrating initial promising results. At the same time, several pitfalls, like the multiple ways of instructing an LLM to generate the desired result, the different perspectives leading the generation (code-based, image-based, grammar-based), and the presence of hallucinations even for the visualization generation task, make their usage less affordable than expected. Following similar initiatives for benchmarking LLMs, this paper copes with the problem of modeling the evaluation of a generated visualization through an LLM. We propose a theoretical evaluation stack, EvaLLM, that decomposes the evaluation effort in its atomic components, characterizes their nature, and provides an overview of how to implement and interpret them. We also designed and implemented an evaluation platform that provides a benchmarking resource for the visualization generation task. The platform supports automatic and manual scoring conducted by multiple assessors to support a fine-grained and semantic evaluation based on the EvaLLM stack. Two case studies on GPT3.5-turbo with Code Interpreter and Llama2-70-b models show the benefits of EvaLLM and illustrate interesting results on the current state-of-the-art LLM-generated visualizations.

Motivation & Objective

• To address the lack of standardized, comprehensive evaluation frameworks for LLM-generated visualizations, which are increasingly used but prone to hallucinations and inconsistent adherence to best practices.
• To model the evaluation process as a decomposable stack of atomic components, enabling structured, interpretable, and repeatable assessment of visualization quality.
• To support both automated and human-in-the-loop evaluation by designing a web-based platform that implements the EvaLLM stack for benchmarking.
• To demonstrate the framework’s utility through empirical case studies on state-of-the-art LLMs, identifying recurring errors in visualization generation.
• To lay the groundwork for future comparative benchmarking and error taxonomy development in LLM-based visualization tasks.

Proposed method

• Propose EvaLLM as a conceptual stack that organizes evaluation into layered components, each representing a distinct dimension of visualization quality (e.g., structural, semantic, perceptual).
• Define multiple levels within each layer to enable fine-grained scoring, such as correctness of data mapping, choice of visualization type, and adherence to design principles.
• Implement a web-based evaluation platform that supports both automated scoring (via LLMs or rule-based checks) and manual labeling by multiple assessors.
• Integrate the platform with the NvBench dataset to evaluate 50 samples across two LLMs: GPT-3.5-turbo with Code Interpreter and Llama2-70b.
• Use the platform to collect and analyze scores across EvaLLM layers, identifying error patterns and evaluating model performance in a standardized way.
• Design the evaluation pipeline to support future expansion to complex visualizations (e.g., Sankey diagrams, geo-maps) and dashboards.

Experimental results

Research questions

• RQ1 How can LLM-generated visualizations be evaluated in a structured, multi-faceted, and interpretable way that captures both structural and semantic quality?
• RQ2 What are the most common types of errors in LLM-generated visualizations, and how can they be systematically categorized and diagnosed using a layered evaluation framework?
• RQ3 To what extent can a standardized evaluation stack like EvaLLM enable fair and quantitative comparison across different LLMs in visualization generation tasks?
• RQ4 How do human assessors and automated systems align in scoring LLM-generated visualizations, and what role does human-in-the-loop evaluation play in detecting subtle semantic flaws?
• RQ5 Can the proposed framework support the development of a taxonomy of LLM errors in visualization generation, and how might this inform model improvement and mitigation strategies?

Key findings

• The case studies revealed that both GPT-3.5-turbo and Llama2-70b frequently generate visualizations with structural flaws, such as incorrect data mapping or inappropriate chart types.
• Semantic errors were prevalent, including misinterpretation of user intent, incorrect labeling, and failure to follow visualization best practices, even when the visual output appeared plausible.
• The evaluation platform successfully supported multi-assessor manual scoring, enabling reliable detection of nuanced issues that automated checks alone could not identify.
• The EvaLLM stack enabled clear identification of error types at specific layers—e.g., perceptual issues in color usage or layout—allowing for targeted model improvement.
• Despite limited sample size (50 from NvBench), the framework demonstrated strong potential for scalable benchmarking and could be extended to complex visualizations like Sankey diagrams and geo-maps.
• The results suggest that current LLMs, while capable of generating basic visualizations, still lack robustness and consistency in adhering to visualization design principles, highlighting the need for standardized evaluation.