[Paper Review] Spot the Difference: Accuracy of Numerical Simulations via the Human Visual System

Kiwon Um, Xiangyu Hu+2 more|arXiv (Cornell University)|Jul 8, 2019

Data Visualization and Analytics47 references4 citations

TL;DR

This paper proposes using crowd-sourced human visual system (HVS) evaluations via a two-alternative forced choice (2AFC) task to assess the accuracy of numerical simulations, especially in under-resolved or near-convergence regimes where traditional metrics fail. It demonstrates that non-experts can reliably rank simulation results, with the T5o scheme showing the highest and most consistent performance across fluid dynamics test cases.

ABSTRACT

Comparative evaluation lies at the heart of science, and determining the accuracy of a computational method is crucial for evaluating its potential as well as for guiding future efforts. However, metrics that are typically used have inherent shortcomings when faced with the under-resolved solutions of real-world simulation problems. We show how to leverage crowd-sourced user studies in order to address the fundamental problems of widely used classical evaluation metrics. We demonstrate that such user studies, which inherently rely on the human visual system, yield a very robust metric and consistent answers for complex phenomena without any requirements for proficiency regarding the physics at hand. This holds even for cases away from convergence where traditional metrics often end up inconclusive results. More specifically, we evaluate results of different essentially non-oscillatory (ENO) schemes in different fluid flow settings. Our methodology represents a novel and practical approach for scientific evaluations that can give answers for previously unsolved problems.

Motivation & Objective

• To address the limitations of classical numerical evaluation metrics like RMSE and PSNR in assessing simulation accuracy, especially in non-converged or under-resolved cases.
• To investigate whether the human visual system (HVS) can serve as a robust, perception-based alternative for evaluating complex fluid simulation results.
• To develop and validate a practical, scalable methodology for scientific evaluation using perceptual judgments from non-expert participants.
• To quantify near-convergence consistency of numerical schemes across varying resolutions, a regime where traditional metrics often fail.
• To demonstrate that perceptual evaluation via HVS can yield reliable, consistent rankings without requiring domain expertise from participants.

Proposed method

• Employs a two-alternative forced choice (2AFC) user study design where participants select which of two simulation images is closer to a reference image.
• Uses the Bradley-Terry model to convert pairwise voting results into a performance score vector, estimating the relative accuracy of each simulation.
• Relies on crowd-sourced participants with no domain expertise, ensuring the evaluation reflects natural human perception.
• Applies the method to evaluate different essentially non-oscillatory (ENO) schemes across multiple fluid dynamics test cases: Taylor-Green vortex and viscous shock tube.
• Introduces the near-convergence consistency metric (NCM) to assess stability and reliability of schemes across resolution levels.
• Uses high-resolution simulations as reference solutions to define the ground truth for perceptual comparison.

Experimental results

Research questions

• RQ1 Can the human visual system provide a more reliable and consistent evaluation of numerical simulation accuracy than classical metrics like RMSE or PSNR in complex, non-converged flow regimes?
• RQ2 To what extent can non-experts accurately rank simulation results based on visual similarity without knowledge of fluid dynamics or numerical methods?
• RQ3 How do different ENO schemes perform in terms of perceptual accuracy and consistency across varying resolution levels?
• RQ4 Can perceptual evaluation via HVS detect meaningful differences in simulation quality that traditional metrics fail to capture?
• RQ5 What is the near-convergence consistency of various schemes, and how does it correlate with perceptual performance?

Key findings

• The T5o scheme achieved the highest mean winning probability (0.829) across all resolutions in the Taylor-Green vortex case, indicating superior perceptual accuracy.
• The W6c scheme, despite its high-order formulation, performed worst in the Taylor-Green vortex case, with a mean winning probability of only 0.086.
• In the viscous shock tube case, the T5o scheme had the highest mean winning probability (0.699), but also the highest standard deviation (0.174), indicating high inconsistency across resolutions.
• The W6c scheme showed the lowest standard deviation (0.063) in the viscous shock tube case, making it the most consistent performer despite lower mean accuracy.
• The NCM metric revealed that T5o is highly accurate but unstable across resolutions, while W6c is less accurate but more reliable, highlighting trade-offs not visible with traditional metrics.
• Crowd-sourced HVS evaluations produced consistent and robust rankings even in non-converged regimes, where classical metrics like RMSE and PSNR often yield inconclusive or misleading results.