[Paper Review] An Evaluation-Focused Framework for Visualization Recommendation Algorithms

Zehua Zeng, Phoebe Moh+6 more|arXiv (Cornell University)|Sep 6, 2021

Data Visualization and Analytics37 references4 citations

TL;DR

This paper proposes an evaluation-focused framework to systematically compare visualization recommendation algorithms by modeling the full design space as a graph, defining traversal strategies, and using oracles for ranking candidates. The framework enables both theoretical and empirical evaluation, revealing that traversal method (BFS vs. DFS) and oracle choice (Dziban vs. CompassQL) significantly influence user preference and exposure to visual designs, though not always user performance.

ABSTRACT

Although we have seen a proliferation of algorithms for recommending visualizations, these algorithms are rarely compared with one another, making it difficult to ascertain which algorithm is best for a given visual analysis scenario. Though several formal frameworks have been proposed in response, we believe this issue persists because visualization recommendation algorithms are inadequately specified from an evaluation perspective. In this paper, we propose an evaluation-focused framework to contextualize and compare a broad range of visualization recommendation algorithms. We present the structure of our framework, where algorithms are specified using three components: (1) a graph representing the full space of possible visualization designs, (2) the method used to traverse the graph for potential candidates for recommendation, and (3) an oracle used to rank candidate designs. To demonstrate how our framework guides the formal comparison of algorithmic performance, we not only theoretically compare five existing representative recommendation algorithms, but also empirically compare four new algorithms generated based on our findings from the theoretical comparison. Our results show that these algorithms behave similarly in terms of user performance, highlighting the need for more rigorous formal comparisons of recommendation algorithms to further clarify their benefits in various analysis scenarios.

Motivation & Objective

• To address the lack of standardized evaluation for visualization recommendation algorithms, which hinders fair comparison and selection.
• To develop a formal framework that specifies algorithms through three components: design space graph, traversal method, and ranking oracle.
• To enable both theoretical comparison and empirical evaluation of algorithm performance across different visual analysis tasks.
• To investigate how algorithmic choices impact user performance, interaction behavior, and preference in data exploration.
• To provide a reusable, open-source evaluation infrastructure for future algorithm development and benchmarking.

Proposed method

• Model the full design space of visualizations as a graph, where nodes represent visual designs and edges represent transformations between them.
• Define traversal strategies (e.g., BFS, DFS) to explore candidate visualizations systematically from a given starting point.
• Use oracles (e.g., CompassQL, Dziban) to rank candidate visualizations based on criteria like statistical interestingness or perceptual distance.
• Conduct a between-subjects user study with 24 participants, each exposed to one algorithm variant, to measure performance on focused and open-ended tasks.
• Apply Bayesian statistical analysis to compare user performance, with 95% credible intervals used to assess significance of differences between algorithm pairs.
• Share source code and a demo video via an OSF repository to ensure reproducibility and facilitate future evaluation.

Experimental results

Research questions

• RQ1 How do different traversal strategies (BFS vs. DFS) affect the diversity of visual designs and data variants exposed to users?
• RQ2 How do different ranking oracles (CompassQL vs. Dziban) influence user performance and preference in visual analysis tasks?
• RQ3 To what extent do algorithmic choices impact user interaction behavior and perceived utility in data exploration?
• RQ4 Can a standardized evaluation framework enable more rigorous and comparable assessment of visualization recommendation algorithms?
• RQ5 How do user preferences and performance vary across different types of visual analysis tasks (focused vs. open-ended)?

Key findings

• BFS traversal exposed significantly more unique variable sets and visual designs than DFS, particularly when combined with Dziban as the oracle.
• Participants significantly preferred BFS over DFS in terms of utility and ease of use, though no significant difference in task performance was found.
• Dziban was preferred over CompassQL in all user rating metrics (efficiency, utility, ease of use, overall), though differences were not statistically significant.
• Dziban outperformed CompassQL in exposing more visual designs and data variants, especially in open-ended tasks, suggesting better exploration support.
• No significant difference in user performance was found between Dziban and CompassQL on focused tasks, despite Dziban’s improved perceptual ranking.
• User preferences varied by task type, indicating that no single algorithm performs optimally across all analysis scenarios, and task-specific algorithm selection may be necessary.