[Paper Review] Encoding Invariances in Deep Generative Models

Viraj Shah, Ameya Joshi+5 more|arXiv (Cornell University)|Jun 4, 2019

Generative Adversarial Networks and Image Synthesis47 references20 citations

TL;DR

This paper introduces InvNet, a deep generative model that encodes known invariances—such as physical laws, geometric motifs, or statistical constraints—into generative adversarial networks (GANs) via an additional invariance loss. By integrating a three-way alternating optimization with extra-gradient descent, InvNet achieves stable training and generates data that strictly obey predefined invariances, outperforming standard GANs and existing PDE solvers in sample efficiency and accuracy.

ABSTRACT

Reliable training of generative adversarial networks (GANs) typically require massive datasets in order to model complicated distributions. However, in several applications, training samples obey invariances that are extit{a priori} known; for example, in complex physics simulations, the training data obey universal laws encoded as well-defined mathematical equations. In this paper, we propose a new generative modeling approach, InvNet, that can efficiently model data spaces with known invariances. We devise an adversarial training algorithm to encode them into data distribution. We validate our framework in three experimental settings: generating images with fixed motifs; solving nonlinear partial differential equations (PDEs); and reconstructing two-phase microstructures with desired statistical properties. We complement our experiments with several theoretical results.

Motivation & Objective

• To address the high sample complexity of standard GANs in modeling complex distributions when prior invariances are known.
• To develop a method that explicitly enforces analytically defined invariances—such as physical laws or statistical constraints—during generative modeling.
• To enable data-efficient, stable training of generative models by integrating invariance constraints into the GAN framework.
• To demonstrate the framework's effectiveness across diverse domains: image motif generation, PDE solution synthesis, and microstructure simulation.

Proposed method

• Proposes InvNet, a GAN-based framework that extends the standard generator-discriminator setup with an Invariance Checker (I(.)) to enforce structural or statistical invariances.
• Introduces a minimax game formulation that includes a novel invariance loss term, ensuring the generator learns to produce samples satisfying predefined constraints.
• Employs a three-way alternating optimization: updating generator, discriminator, and invariance loss in sequence to stabilize training.
• Uses extra-gradient descent (EGD) to stabilize training dynamics, proven to converge where standard gradient descent diverges.
• Applies moment-matching constraints to encode statistical invariances in microstructure generation.
• Designs a modified discriminator architecture that better captures and enforces invariances, improving training stability and performance.

Experimental results

Research questions

• RQ1 Can a deep generative model be systematically extended to enforce known invariances such as PDEs or geometric motifs without requiring large datasets?
• RQ2 How does the choice of discriminator architecture influence the ability of the model to encode target invariances in the generator?
• RQ3 Why do standard gradient descent updates in InvNet lead to unstable training dynamics, and how can this be remedied?
• RQ4 Can InvNet achieve competitive performance in solving PDEs without any training data, relying solely on structural invariance?
• RQ5 To what extent does enforcing invariances improve sample quality and reduce sample complexity in generative modeling?

Key findings

• InvNet successfully generates images with fixed motifs by enforcing invariance constraints, demonstrating that invariance-aware training improves fidelity and consistency.
• For PDE solving, InvNet achieves competitive results on the nonlinear Burgers’ equation in both viscous and inviscid regimes, even when trained without any data—only using the PDE as an invariance.
• In microstructure generation, InvNet outperforms standard numerical methods in computational efficiency while accurately matching target statistical properties.
• Theoretical analysis shows that standard gradient descent diverges in InvNet training due to singular values of the update matrix exceeding 1, necessitating alternative optimization strategies.
• Extra-gradient descent (EGD) is proven to stabilize training and form a contractive mapping, enabling convergence to the correct equilibrium under proper step size selection.
• When the true data distribution satisfies the invariance, the discriminator learns to incorporate it; otherwise, the discriminator must be modified to avoid invariance mismatch.