[Paper Review] viz2viz: Prompt-driven stylized visualization generation using a diffusion model

Jiaqi Wu, John Joon Young Chung+1 more|arXiv (Cornell University)|Apr 4, 2023

Data Visualization and Analytics4 citations

TL;DR

viz2viz introduces a prompt-driven, diffusion model-based framework for generating stylized visualizations by transforming basic charts (e.g., bar, pie, area, network charts) into photorealistic or artistic forms using text prompts. It uses a three-step recipe—sketch, synthesis, and refine—combined with controllable diffusion pipelines (e.g., ControlNet, img2img, DMP) to preserve data fidelity while enabling high visual coherence and creative flexibility.

ABSTRACT

Creating stylized visualization requires going beyond the limited, abstract, geometric marks produced by most tools. Rather, the designer builds stylized idioms where the marks are both transformed (e.g., photographs of candles instead of bars) and also synthesized into a 'scene' that pushes the boundaries of traditional visualizations. To support this, we introduce viz2viz, a system for transforming visualizations with a textual prompt to a stylized form. The system follows a high-level recipe that leverages various generative methods to produce new visualizations that retain the properties of the original dataset. While the base recipe is consistent across many visualization types, we demonstrate how it can be specifically adapted to the creation of different visualization types (bar charts, area charts, pie charts, and network visualizations). Our approach introduces techniques for using different prompts for different marks (i.e., each bar can be something completely different) while still retaining image "coherence." We conclude with an evaluation of the approach and discussion on extensions and limitations.

Motivation & Objective

• To address the lack of tools that enable designers to create stylized visualizations—where abstract marks (e.g., bars, slices) are replaced with complex, realistic, or artistic representations (e.g., candles, coffee cups, flowers) while preserving data semantics.
• To overcome limitations of existing generative models (e.g., DALL-E2, Stable Diffusion) that fail to maintain data correspondence when generating stylized visualizations from text prompts.
• To develop a flexible, reusable workflow that decouples mark generation from scene composition, enabling consistent application across multiple visualization types without retraining.
• To ensure visual coherence and data fidelity in the final output, even when marks are replaced with semantically complex objects (e.g., syringes for data values, buildings for bars).

Proposed method

• The system uses a three-stage recipe: (1) sketch—generating a masked, simplified version of the input visualization to preserve structure; (2) synthesis—applying text prompts to generate stylized marks using diffusion models (e.g., Stable Diffusion) with control mechanisms like ControlNet and depth2Img; (3) refine—using inpainting and post-processing to correct inconsistencies and enhance coherence.
• A custom pipeline, DMP (Diffusion Mark Prompting), is introduced to guide the diffusion process by conditioning on both the original mark’s shape and the desired stylized object via text prompts, enabling precise control over mark transformation.
• The method leverages existing controllable diffusion models (e.g., img2img, ControlNet) to maintain spatial and structural alignment between the original data and the stylized output, ensuring that data proportions (e.g., pie slice sizes, bar heights) are preserved.
• Image processing techniques such as masking, translation, and blending are used to isolate and transform individual marks before reassembling them into a cohesive visualization.
• Negative prompts and parameter tuning (e.g., beta in DMP) are used in post-processing to improve image quality and reduce artifacts.
• The approach is generalized across visualization types by adapting the mark-deconstruction and reassembly process for different chart semantics (e.g., bars as stacks of cans, nodes as flowers in a network).

Experimental results

Research questions

• RQ1 Can a diffusion model be effectively guided to transform abstract, data-driven visual marks into complex, stylized, and realistic representations (e.g., candles, coffee cups) while preserving the underlying data distribution?
• RQ2 How can visual coherence be maintained when replacing individual marks with semantically rich, non-geometric objects (e.g., a roller coaster instead of a bar) across multiple marks in a single visualization?
• RQ3 To what extent can a single, reusable workflow (the viz2viz recipe) be adapted across diverse visualization types (bar, area, pie, network charts) without requiring retraining or domain-specific fine-tuning?
• RQ4 What role do intermediate control mechanisms (e.g., ControlNet, depth maps, masks) play in balancing creative freedom with data fidelity in generative visualization?
• RQ5 What are the key failure modes and limitations of prompt-driven stylized visualization generation, and how can they be mitigated using post-processing and parameter tuning?

Key findings

• The viz2viz framework successfully generates stylized visualizations across multiple chart types (bar, area, pie, network) while maintaining data fidelity, as demonstrated through qualitative examples showing correct proportional representation (e.g., pie slices matching data fractions).
• The three-stage recipe—sketch, synthesis, refine—enables high visual coherence even when marks are replaced with complex objects (e.g., coffee cups, tulips, cityscapes), with results showing minimal distortion of data structure.
• The use of DMP (Diffusion Mark Prompting) allows for targeted, prompt-driven transformation of individual marks while preserving spatial relationships, significantly improving control over the generative process compared to end-to-end diffusion.
• On average, achieving a high-quality output required around 10 generations per prompt, indicating that prompt quality and parameter tuning are critical, though the system’s default configurations and iterative refinement steps improve success rates.
• The system can fail partially or completely, especially when the visual or color contrast between the original mark and the desired object is high (e.g., yellow pie slice → red pizza slice), but post-generation inpainting can effectively recover from such failures.
• Limitations include difficulty in generating certain infomages (e.g., top-hat in Figure 2) due to poor foreground-background interaction, and challenges in scaling to large numbers of marks due to blending artifacts during diffusion.