[Paper Review] Physics-Informed Computer Vision: A Review and Perspectives

Chayan Banerjee, Kien Nguyen+2 more|arXiv (Cornell University)|May 29, 2023

Cell Image Analysis Techniques4 citations

TL;DR

This paper introduces a systematic taxonomy for physics-informed computer vision (PICV), integrating physical laws into computer vision pipelines through data, network architecture, or loss function modifications. It reviews how physical priors—such as fluid dynamics, electromagnetic models, and kinematic constraints—enhance model robustness, data efficiency, and physical plausibility across tasks like 3D reconstruction, human motion analysis, and video forecasting.

ABSTRACT

The incorporation of physical information in machine learning frameworks is opening and transforming many application domains. Here the learning process is augmented through the induction of fundamental knowledge and governing physical laws. In this work, we explore their utility for computer vision tasks in interpreting and understanding visual data. We present a systematic literature review of more than 250 papers on formulation and approaches to computer vision tasks guided by physical laws. We begin by decomposing the popular computer vision pipeline into a taxonomy of stages and investigate approaches to incorporate governing physical equations in each stage. Existing approaches in computer vision tasks are analyzed with regard to what governing physical processes are modeled and formulated, and how they are incorporated, i.e. modification of input data (observation bias), modification of network architectures (inductive bias), and modification of training losses (learning bias). The taxonomy offers a unified view of the application of the physics-informed capability, highlighting where physics-informed learning has been conducted and where the gaps and opportunities are. Finally, we highlight open problems and challenges to inform future research. While still in its early days, the study of physics-informed computer vision has the promise to develop better computer vision models that can improve physical plausibility, accuracy, data efficiency, and generalization in increasingly realistic applications.

Motivation & Objective

• To address the limitations of purely data-driven computer vision models, which often lack physical plausibility, interpretability, and generalization under sparse or noisy data.
• To establish a unified framework for incorporating physical laws into computer vision by decomposing the CV pipeline into stages where physics can be systematically integrated.
• To identify gaps in current PICV applications, particularly in classification, segmentation, and crowd analysis, where physical priors remain underutilized.
• To provide a comprehensive review of physics-informed approaches across imaging, human motion, robotics, and 3D reconstruction, highlighting effective strategies and open challenges.
• To advocate for standardized benchmarking platforms to enable fair evaluation and comparison of PICV methods across diverse domains.

Proposed method

• Proposes a taxonomy of physics-informed computer vision (PICV) based on three integration strategies: observation bias (modifying input data), inductive bias (modifying network architecture), and learning bias (modifying loss functions).
• Classifies physics priors by type—e.g., Navier-Stokes for fluid dynamics, Maxwell’s equations for imaging, geometric constraints for 3D reconstruction, and kinematic limits for human motion.
• Analyzes how physical laws are encoded: as explicit equations in loss functions (e.g., PDE-constrained training), as regularizers, or as auxiliary network inputs.
• Reviews case studies across domains: NERF-based 3D reconstruction using geometric priors, human pose estimation with anatomical joint limits, and crowd motion modeling via entropy and order parameters.
• Integrates insights from physics-informed machine learning (PIML) into CV, adapting PDE-constrained neural networks and physics-guided generative models to visual data modalities.
• Uses a pipeline-based framework to map physical priors to specific CV stages (e.g., feature extraction, regression, generation), enabling systematic analysis of integration points.

Figure 1 . (a) Timeline of PICV papers published over the last five years, where the histogram presents a exponentially increasing trend, (b) Application domains of recent PICV papers. The most applied domain is fluid and solid mechanics closely followed by imaging and photonics applications. While

Experimental results

Research questions

• RQ1 How can physical laws be systematically incorporated into different stages of the computer vision pipeline to improve model performance and physical consistency?
• RQ2 What types of physical priors (e.g., fluid dynamics, electromagnetic models, kinematic constraints) are most effective across different computer vision tasks?
• RQ3 In what ways do observation bias, inductive bias, and learning bias differ in their impact on model accuracy, data efficiency, and generalization in vision tasks?
• RQ4 Why is the adoption of physics-informed approaches limited in tasks like image classification, semantic segmentation, and crowd analysis despite their potential?
• RQ5 What are the key challenges in benchmarking and evaluating PICV models due to domain-specific datasets and lack of standardized platforms?

Key findings

• The number of PICV papers has grown exponentially over the past five years, with the highest adoption in fluid and solid mechanics, followed by imaging and photonics applications.
• Physics-informed models improve data efficiency and generalization, particularly in high-dimensional or data-scarce scenarios such as 3D reconstruction and human motion analysis.
• In human motion tracking, incorporating physical constraints like joint limits and kinematic consistency enhances pose estimation accuracy and temporal coherence.
• Physics-informed loss functions, such as those enforcing PDEs (e.g., Navier-Stokes or heat equations), lead to more physically plausible predictions in video forecasting and super-resolution tasks.
• Despite progress, significant research gaps remain in classification, segmentation, and crowd analysis, where physical priors are underutilized compared to generative or forecasting tasks.
• A lack of standardized benchmarking platforms hinders fair comparison and reproducibility of PICV methods, limiting the field’s advancement and adoption.

Figure 2 . A simplified illustrative example of physics incorporation in a CV task, adapted from (Behera et al . , 2021 ) . Here the physics in the form of flow information is extracted from the data/ video segment sequence and incorporated in an aggregating network (PIDLNet).