[Paper Review] Not As Easy As You Think -- Experiences and Lessons Learnt from Trying to Create a Bottom-Up Visualization Image Typology

Jian Chen, Petra Isenberg+5 more|arXiv (Cornell University)|Sep 15, 2022

Data Visualization and Analytics4 citations

TL;DR

This paper presents a novel, bottom-up typology of 13 visualization image types derived from a two-year qualitative analysis of 6,833 figures in IEEE VIS papers (1990–2020). Despite initial assumptions of simplicity, the coding process revealed significant perceptual ambiguity and interpretive variation, leading to a robust, community-validated typology that supports research, teaching, and future standardization in visualization.

ABSTRACT

We present and discuss the results of a two-year qualitative analysis of images published in IEEE Visualization (VIS) papers. Specifically, we derive a typology of 13 visualization image types, coded to distinguish visualizations and several image characteristics. The categorization process required much more time and was more difficult than we initially thought. The resulting typology and image analysis may serve a number of purposes: to study the evolution of the community and its research output over time, to facilitate the categorization of visualization images for the purpose of teaching, to identify visual designs for evaluation purposes, or to enable progress towards standardization in visualization. In addition to the typology and image characterization, we provide a dataset of 6,833 tagged images and an online tool that can be used to explore and analyze the large set of tagged images. We thus facilitate a discussion of the diverse visualizations used and how they are published and communicated in our community.

Motivation & Objective

• To develop a comprehensive, bottom-up typology of visualization images based on visual appearance rather than data or intent.
• To understand the challenges and complexities involved in categorizing visualization images in real-world academic publications.
• To support future research in visualization by enabling systematic analysis of visual output, teaching, and image retrieval.
• To identify recurring visual design patterns and their evolution across 30 years of IEEE VIS publications.
• To provide a dataset and online tool for exploring and analyzing visualization image types in the visualization research community.

Proposed method

• Conducted a two-year qualitative coding process on 6,833 figures from 695 IEEE VIS papers spanning 1990 to 2020.
• Developed a typology of 13 visualization image types based on visual design, dimensionality, and structural characteristics, not data semantics or author intent.
• Used iterative consensus among expert coders to resolve ambiguities and refine categories through extensive discussion.
• Classified images based on visual appearance (e.g., 2D schematic, 3D surface/volume, line-based, GUI, glyph, grid) and added metadata such as dimensionality and perceptual ambiguity.
• Generated a dataset of 6,833 tagged images and built an online tool for interactive exploration and analysis.
• Validated the typology through internal consistency checks and iterative refinement over two years, acknowledging limitations in coder agreement and perceptual uncertainty.

Experimental results

Research questions

• RQ1 What are the most prevalent visualization image types in IEEE VIS publications across three decades?
• RQ2 How difficult is it to consistently categorize visualization images based solely on visual appearance, and what factors contribute to coding ambiguity?
• RQ3 To what extent do expert coders agree on image categorization, and how does prior expertise influence classification accuracy?
• RQ4 Can a bottom-up typology of visualization images serve as a foundation for research, teaching, and standardization in visualization?
• RQ5 How do visual design trends and image types evolve over time in visualization research publications?

Key findings

• The most common visualization image types are 2D schematics (1,432 images), 3D surface/volume renderings (655), line-based techniques (643), and GUIs (411), collectively representing 73% of the 6,833 analyzed images.
• Despite shared expertise, coders experienced significant disagreement on ambiguous cases, especially for volume rendering, glyphs, and grid-based techniques, indicating perceptual and interpretive uncertainty.
• The coding process proved far more complex and time-consuming than anticipated, with many images defying clear categorization due to hybrid designs and lack of standardized definitions.
• Expert coders struggled to distinguish between similar techniques (e.g., volume vs. surface rendering), and even domain experts made errors when images were unfamiliar.
• The resulting typology of 13 image types provides a stable, community-validated framework for classifying visualization output based on visual appearance.
• The dataset of 6,833 tagged images and associated online tool enable detailed exploration of visual design trends and support future research in image retrieval and visualization analysis.