[论文解读] Unsupervised Object Representation Learning using Translation and Rotation Group Equivariant VAE

Alireza Nasiri, Tristan Bepler|arXiv (Cornell University)|Oct 24, 2022

Domain Adaptation and Few-Shot Learning被引用 7

一句话总结

TARGET-VAE 是一种完全无监督的变分自编码器，通过在编码器和生成器网络中强制实施群等变性，学习解耦的、旋转和位移不变的对象表征。它联合推断语义、旋转和位移潜在变量，在严重损坏的图像上也实现了无监督姿态估计和语义聚类的最先进性能。

ABSTRACT

In many imaging modalities, objects of interest can occur in a variety of locations and poses (i.e. are subject to translations and rotations in 2d or 3d), but the location and pose of an object does not change its semantics (i.e. the object's essence). That is, the specific location and rotation of an airplane in satellite imagery, or the 3d rotation of a chair in a natural image, or the rotation of a particle in a cryo-electron micrograph, do not change the intrinsic nature of those objects. Here, we consider the problem of learning semantic representations of objects that are invariant to pose and location in a fully unsupervised manner. We address shortcomings in previous approaches to this problem by introducing TARGET-VAE, a translation and rotation group-equivariant variational autoencoder framework. TARGET-VAE combines three core innovations: 1) a rotation and translation group-equivariant encoder architecture, 2) a structurally disentangled distribution over latent rotation, translation, and a rotation-translation-invariant semantic object representation, which are jointly inferred by the approximate inference network, and 3) a spatially equivariant generator network. In comprehensive experiments, we show that TARGET-VAE learns disentangled representations without supervision that significantly improve upon, and avoid the pathologies of, previous methods. When trained on images highly corrupted by rotation and translation, the semantic representations learned by TARGET-VAE are similar to those learned on consistently posed objects, dramatically improving clustering in the semantic latent space. Furthermore, TARGET-VAE is able to perform remarkably accurate unsupervised pose and location inference. We expect methods like TARGET-VAE will underpin future approaches for unsupervised object generation, pose prediction, and object detection.

研究动机与目标

• 在无监督条件下学习对图像中平移和旋转不变的语义对象表征。
• 解决标准 VAE 在解耦姿态和位置与语义内容方面的局限性。
• 通过引入空间变换的结构归纳偏置，改进无监督姿态和位置推理。
• 在冷冻电镜等成像模态中实现鲁棒的表征学习，其中对象随机取向和定位。
• 开发一个完全可微、端到端的框架，联合优化解耦和重建。

提出的方法

• 采用旋转和位移群等变的编码器网络，以提取与变换相关的特征。
• 使用结构解耦的变分后验分布，将潜在变量分离为语义、旋转和位移分量。
• 实现一个空间等变的生成器网络，从解耦的潜在变量重建图像。
• 在旋转上应用均匀先验，并使用联合推理网络近似所有潜在变量的后验分布。
• 通过在编码器和生成器中共享权重和使用群卷积操作来强制实现等变性。
• 仅使用观测图像端到端训练整个模型，姿态和位置无任何监督信号。

实验结果

研究问题

• RQ1 完全无监督的 VAE 是否能够学习到对物体平移和旋转不变的解耦表征？
• RQ2 该模型是否能在无任何监督的情况下准确推断物体姿态（旋转和平移）？
• RQ3 解耦表征是否能提升下游语义物体类别聚类的性能？
• RQ4 该框架是否能泛化到高噪声和高变异性的现实成像数据（如冷冻电镜显微图像）？
• RQ5 与标准 VAE 相比，群等变设计如何提升表征质量？

主要发现

• 即使在训练图像因旋转和平移严重损坏的情况下，TARGET-VAE 学习到的语义表征仍接近于一致姿态数据的准确度。
• 该模型实现了高质量的无监督姿态估计，无需任何监督即可准确推断物体的旋转和平移。
• 潜在空间中的语义聚类与已知的语义标签一致，冷冻电镜数据集中粒子视角和污染物被清晰分离。
• 在 EMPIAR-10025 上，TARGET-VAE 识别出三个对应于不同粒子视角和污染状态的独立聚类。
• 在 EMPIAR-10029 上，该模型发现了粒子视角的连续变化，生成了与旋转和平移无关的去噪重建图像。
• 该框架在解耦性方面显著优于先前方法，并避免了姿态-语义纠缠等常见病理性问题。