[论文解读] Improving Disentangled Representation Learning with the Beta Bernoulli Process

Prashnna Gyawali, Zhiyuan Li+5 more|arXiv (Cornell University)|Sep 3, 2019

Adversarial Robustness in Machine Learning参考文献 39被引用 4

一句话总结

该论文提出IBP-VAE，一种变分自编码器，采用印度餐厅套餐过程（IBP）作为非参数先验，以增强解耦表征学习。通过在保持因子独立性的同时提升后验建模能力，IBP-VAE在复杂数据上实现了更优的解耦效果，在MNIST、3D Chairs、dSprites以及临床心电图和皮肤病变数据集上均优于当前最先进方法，实现了对干扰因子的更好解耦与下游任务准确率的提升。

ABSTRACT

To improve the ability of VAE to disentangle in the latent space, existing works mostly focus on enforcing independence among the learned latent factors. However, the ability of these models to disentangle often decreases as the complexity of the generative factors increases. In this paper, we investigate the little-explored effect of the modeling capacity of a posterior density on the disentangling ability of the VAE. We note that the independence within and the complexity of the latent density are two different properties we constrain when regularizing the posterior density: while the former promotes the disentangling ability of VAE, the latter -- if overly limited -- creates an unnecessary competition with the data reconstruction objective in VAE. Therefore, if we preserve the independence but allow richer modeling capacity in the posterior density, we will lift this competition and thereby allow improved independence and data reconstruction at the same time. We investigate this theoretical intuition with a VAE that utilizes a non-parametric latent factor model, the Indian Buffet Process (IBP), as a latent density that is able to grow with the complexity of the data. Across three widely-used benchmark data sets and two clinical data sets little explored for disentangled learning, we qualitatively and quantitatively demonstrated the improved disentangling performance of IBP-VAE over the state of the art. In the latter two clinical data sets riddled with complex factors of variations, we further demonstrated that unsupervised disentangling of nuisance factors via IBP-VAE -- when combined with a supervised objective -- can not only improve task accuracy in comparison to relevant supervised deep architectures but also facilitate knowledge discovery related to task decision-making. A shorter version of this work will appear in the ICDM 2019 conference proceedings.

研究动机与目标

• 为解决现有VAE在受限后验建模能力下难以解耦复杂生成因子的局限性。
• 探究VAE中后验独立性与建模能力之间的权衡，特别是建模能力受限如何损害解耦性能。
• 通过非参数先验提升后验建模能力，同时保持因子独立性，以改善解耦效果。
• 在临床数据中验证IBP-VAE在无监督解耦干扰因子方面的有效性，提升可解释性与下游任务性能。
• 在基准数据集与尚未充分探索的临床数据集上验证该方法，这些数据集具有复杂且高维的变异。

提出的方法

• IBP-VAE采用印度餐厅套餐过程（IBP）作为非参数先验，以建模无限多个独立潜在因子，从而实现灵活的后验建模能力。
• IBP先验使模型能随数据增长而增加复杂度，避免因后验过度受限而引发的重建与解耦目标之间的冲突。
• 使用指数族分布（Concrete和Kumaraswamy分布）对IBP中的伯努利和贝塔变量进行可微分近似，支持通过反向传播实现端到端训练。
• 模型采用标准VAE目标函数并加入KL散度正则化，但通过先验结构确保因子独立性，同时支持更丰富的后验表征。
• 针对临床数据，引入条件IBP-VAE（cIBP-VAE），以解耦任务相关因子与干扰因子，如心电图信号中的起搏伪影。
• 模型架构采用卷积编码器与解码器，配合批量归一化和ReLU激活函数，适配图像与临床信号数据。

Figure 1: Outline of the presented IBP-VAE (blue) for unsupervised disentangled representation learning, and cIBP-VAE (blue and orange) for the combination with supervised task learning.

实验结果

研究问题

• RQ1 在生成因子复杂的情况下，是否增加VAE后验密度的建模能力可提升解耦性能？
• RQ2 像IBP这样的非参数先验能否在保持因子独立性的同时，通过支持更丰富的后验表征来缓解重建与解耦目标之间的冲突？
• RQ3 IBP-VAE在具有多样化生成因子的基准数据集上的表现如何，相较于当前最先进解耦VAE方法？
• RQ4 通过IBP-VAE实现无监督干扰因子解耦，是否能提升在高变异临床数据中的监督任务性能？
• RQ5 IBP-VAE在真实世界临床数据中，能在多大程度上揭示并解耦出可解释、语义上有意义的因子？

主要发现

• 在MNIST、3D Chairs和dSprites数据集上，IBP-VAE实现了最先进水平的解耦性能，在定性与定量评估中均优于β-VAE和β-TCVAE等方法。
• 在dSprites数据集中，IBP-VAE成功解耦了旋转与尺度等连续因子，解耦度量中清晰显示出优异的解耦效果。
• 在心电图数据集上，cIBP-VAE在重建无起搏伪影方面显著优于c-VAE（p < 0.01），证明其对复杂干扰因子的建模能力更强。
• cIBP-VAE成功解耦了起搏伪影因子，表现为通过样本间交换表征，可在重建中转移伪影的有无。
• 在cIBP-VAE中关闭触发单元后，重建信号中引入了起搏伪影，证实该特定干扰因子被成功解耦。
• 在临床皮肤病变分析中，cIBP-VAE在重建准确率上与c-VAE相当，但对与伪影相关的因子实现了更优的解耦，从而提升了可解释性与任务性能。

Figure 2: [Best viewed in color] (a)-(e): Images generated by traversal along a single latent unit (over a range of [-3, 3]) on the latent representation encoded from a random sample (each row). (f): Triggering capacity of the IBP-VAE: column one: original images; column two: reconstructed images; c