[Paper Review] Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discovery

Nathaniel Trask, Carianne Martinez+2 more|arXiv (Cornell University)|Feb 7, 2022

Machine Learning in Materials Science4 citations

TL;DR

This paper proposes Physics-informed Multimodal Autoencoders (PIMA), an unsupervised variational inference framework that disentangles multimodal scientific data by fusing low-fidelity, high-throughput measurements with physics-informed priors. By combining a product of experts for multimodal posterior fusion, a Gaussian mixture prior for shared latent clusters, and a physics-informed mixture of experts for structured disentanglement, PIMA enables accurate cross-modal generation—demonstrated by predicting stress-strain curves from lattice topology images with 94.74% classification accuracy.

ABSTRACT

We introduce physics-informed multimodal autoencoders (PIMA) - a variational inference framework for discovering shared information in multimodal scientific datasets representative of high-throughput testing. Individual modalities are embedded into a shared latent space and fused through a product of experts formulation, enabling a Gaussian mixture prior to identify shared features. Sampling from clusters allows cross-modal generative modeling, with a mixture of expert decoder imposing inductive biases encoding prior scientific knowledge and imparting structured disentanglement of the latent space. This approach enables discovery of fingerprints which may be detected in high-dimensional heterogeneous datasets, avoiding traditional bottlenecks related to high-fidelity measurement and characterization. Motivated by accelerated co-design and optimization of materials manufacturing processes, a dataset of lattice metamaterials from metal additive manufacturing demonstrates accurate cross modal inference between images of mesoscale topology and mechanical stress-strain response.

Motivation & Objective

• To enable unsupervised discovery of scientific fingerprints in high-dimensional, heterogeneous multimodal datasets without costly labeling.
• To overcome bottlenecks in high-throughput materials discovery by replacing slow, high-fidelity measurements with fast, low-fidelity surrogates.
• To integrate physical laws and scientific priors into deep generative models to achieve structured, interpretable disentanglement of latent representations.
• To enable cross-modal generative modeling—inferring high-fidelity modalities from low-fidelity ones—using shared latent clusters.
• To accelerate materials design and process optimization by fusing experimental images with physics-based models in a unified variational inference framework.

Proposed method

• Uses unimodal encoders to map individual modalities $X_i$ into Gaussian posterior distributions $q(Z|X_i)$ in a shared latent space.
• Applies a product of experts (PoE) formulation to fuse unimodal posteriors into a multimodal posterior $q(Z|\mathbf{X}) = \prod_i q(Z|X_i)$, enabling complementary information integration.
• Imposes a Gaussian mixture prior $p(Z|c)$ on the latent space to identify shared, cross-modal clusters $c$ representing scientific fingerprints.
• Employs a physics-informed mixture of experts (MoE) decoder $p(X_i|c,Z)$, where each expert is a parameterized physical model (e.g., linear strain-hardening model), embedding scientific inductive biases.
• Optimizes the model via a novel expectation-maximization strategy on a variational lower bound (ELBO) with closed-form integrals for efficient training.
• Enables cross-modal inference by training unimodal encoders to reconstruct the multimodal posterior, allowing estimation $p(c|X_i)$ for any modality $X_i$.

Experimental results

Research questions

• RQ1 Can an unsupervised deep generative model discover shared, interpretable representations across heterogeneous scientific modalities without labeled data?
• RQ2 How can physics-informed inductive biases be embedded into a multimodal autoencoder to improve disentanglement and interpretability?
• RQ3 Can low-fidelity, high-throughput measurements (e.g., images) be used to accurately infer high-fidelity, slow-to-measure physical responses (e.g., stress-strain curves)?
• RQ4 To what extent does the integration of physical models via a mixture of experts improve generative performance and scientific fidelity compared to purely data-driven approaches?
• RQ5 Can the learned latent clusters (fingerprints) be used to generate new, physically plausible data across modalities?

Key findings

• The PIMA framework achieved 94.74% classification accuracy in identifying latent clusters for stress-strain curves using only image data, demonstrating strong cross-modal inference capability.
• The model successfully generated microstructural images corresponding to distinct mechanical behaviors (octet and gyroid lattices), showing effective generative modeling from latent clusters.
• The use of a physics-informed MoE decoder—specifically a linear strain-hardening model—provided a physically interpretable fit to stress-strain data, capturing key material properties like yield stress and moduli.
• The framework enabled accurate generative modeling of high-fidelity mechanical responses from low-fidelity imaging data, reducing reliance on costly experimental characterization.
• The unsupervised clustering of latent space via a Gaussian mixture prior revealed natural groupings in the data corresponding to distinct material topologies and mechanical responses.
• The method demonstrated feasibility for high-throughput scientific discovery by enabling surrogate modeling of expensive measurements using fast, accessible data sources.