[Paper Review] Towards fully covariant machine learning

Soledad Villar, David W. Hogg+3 more|arXiv (Cornell University)|Jan 31, 2023

Computational Physics and Python Applications9 citations

TL;DR

The paper introduces passive (covariance) and active symmetries, arguing ML models should respect passive symmetries arising from data representations, and discusses implications for generalization, causality, and model design.

ABSTRACT

Any representation of data involves arbitrary investigator choices. Because those choices are external to the data-generating process, each choice leads to an exact symmetry, corresponding to the group of transformations that takes one possible representation to another. These are the passive symmetries; they include coordinate freedom, gauge symmetry, and units covariance, all of which have led to important results in physics. In machine learning, the most visible passive symmetry is the relabeling or permutation symmetry of graphs. Our goal is to understand the implications for machine learning of the many passive symmetries in play. We discuss dos and don'ts for machine learning practice if passive symmetries are to be respected. We discuss links to causal modeling, and argue that the implementation of passive symmetries is particularly valuable when the goal of the learning problem is to generalize out of sample. This paper is conceptual: It translates among the languages of physics, mathematics, and machine-learning. We believe that consideration and implementation of passive symmetries might help machine learning in the same ways that it transformed physics in the twentieth century.

Motivation & Objective

• Define passive and active symmetries in ML and physics contexts.
• Explain how passive symmetries arise from data representation choices (coordinates, units, etc.).
• Provide conceptual guidance and examples showing how enforcing passive symmetries can improve learning and generalization.
• Discuss connections between passive symmetries and causal modeling.
• Offer practical guidance on structuring ML models to respect passive symmetries.

Proposed method

• Formal definitions of passive vs. active symmetries using group actions and commutative diagrams.
• Discussion of units covariance as a universal passive symmetry with dimensional analysis arguments.
• Toy examples illustrating the benefits of enforcing passive symmetries in regressions.
• Conceptual links between passive symmetries and causality, and implications for model design and normalization.
• Glossary translating physics and ML terminology to aid cross-disciplinary understanding.

Experimental results

Research questions

• RQ1 What are passive and active symmetries, and how do they relate to ML representations and data analysis?
• RQ2 How can enforcing passive symmetries (covariances) affect learning outcomes and out-of-sample generalization?
• RQ3 What are the practical challenges in implementing passive symmetries in ML models, and when can they be converted from active symmetries?
• RQ4 How do passive symmetries connect to causal modeling and intervention concepts?
• RQ5 What guidance can be given for data normalization and model architecture to respect passive symmetries?

Key findings

• Passive symmetries arise from representation choices (coordinates, units, gauge, reparameterizations) and are exact by definition.
• Enforcing passive symmetries can reveal underlying scaling laws and missing elements in a problem.
• Units covariance can improve out-of-sample generalization and guide discovery of dimensionful constants (e.g., Planck constant in black-body radiation toy example).
• Many ML practices do not respect passive symmetries, which can lead to systematic mistakes; respecting them suggests changes in regularization, architecture, and normalization.
• There are meaningful connections between passive symmetries and causal inference, including consistency constraints on causal graphs and the role of interventions for identifying necessary inputs.