[论文解读] Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses

Eslam Abdelaleem, Ilya Nemenman+1 more|arXiv (Cornell University)|Oct 5, 2023

Adversarial Robustness in Machine LearningComputer Science被引用 3

一句话总结

该论文提出了深度变分多变量信息瓶颈（DVMIB）框架，这是一种统一的变分降维方法，通过编码器-解码器图结构建模信息压缩。该方法推广了现有方法（如DVCCA和VAE），推导出新算法（如beta-DVCCA和DVSIB），并在Noisy MNIST数据集上展现出更优的样本效率和分类准确率，尤其当结构假设与数据匹配时表现更佳。

ABSTRACT

Variational dimensionality reduction methods are widely used for their accuracy, generative capabilities, and robustness. We introduce a unifying framework that generalizes both such as traditional and state-of-the-art methods. The framework is based on an interpretation of the multivariate information bottleneck, trading off the information preserved in an encoder graph (defining what to compress) against that in a decoder graph (defining a generative model for data). Using this approach, we rederive existing methods, including the deep variational information bottleneck, variational autoencoders, and deep multiview information bottleneck. We naturally extend the deep variational CCA (DVCCA) family to beta-DVCCA and introduce a new method, the deep variational symmetric information bottleneck (DVSIB). DSIB, the deterministic limit of DVSIB, connects to modern contrastive learning approaches such as Barlow Twins, among others. We evaluate these methods on Noisy MNIST and Noisy CIFAR-100, showing that algorithms better matched to the structure of the problem like DVSIB and beta-DVCCA produce better latent spaces as measured by classification accuracy, dimensionality of the latent variables, sample efficiency, and consistently outperform other approaches under comparable conditions. Additionally, we benchmark against state-of-the-art models, achieving superior or competitive accuracy. Our results demonstrate that this framework can seamlessly incorporate diverse multi-view representation learning algorithms, providing a foundation for designing novel, problem-specific loss functions.

研究动机与目标

• 将基于多变量信息瓶颈的多样化变分降维方法统一于一个基于原理的框架下。
• 通过调整编码器和解码器图结构，实现新方法的系统性推导。
• 通过使模型归纳偏差与数据结构相匹配，提升样本效率和表征质量。
• 提供一个模块化、可扩展的框架，利用领域知识设计特定问题的变分损失函数。

提出的方法

• 该框架基于多变量信息瓶颈原理，其中编码器图结构定义压缩过程，解码器图结构定义生成建模过程。
• 使用变分界近似信息瓶颈损失，支持使用深度神经网络进行端到端训练。
• 通过一个beta参数控制压缩与重构之间的权衡，将DVCCA等方法推广为beta-DVCCA。
• 通过定义特定图结构推导新模型：DVSIB通过压缩两个变量以保留其表征间的互信息。
• 该框架支持共享和私有潜在表征，支持如DVSIB-private等模型。
• 通过标准基准（包括Noisy MNIST）实现并评估算法，分类准确率和潜在维度作为关键指标。

Figure 2: Dataset consisting of pairs of digits drawn from MNIST that share an identity. Top row, $X$ : MNIST digits randomly scaled $(0.5-1.5)$ and rotated $(0-\pi/2)$ . Bottom row, $Y$ : MNIST digits with a background Perlin noise. t-SNE of $X$ and $Y$ datasets (left and middle) shows poor separat

实验结果

研究问题

• RQ1 如何系统性地将现有变分降维方法统一于单一信息论框架之下？
• RQ2 在DVMIB框架内，通过修改编码器和解码器图结构，可以推导出哪些新方法？
• RQ3 当模型的归纳偏差（如对称压缩）与数据的潜在结构相匹配时，对表征质量有何影响？
• RQ4 该框架在低数据场景下在多大程度上提升了样本效率？
• RQ5 该框架能否用于推导并实现用于多视图表示学习的新型、特定问题的损失函数？

主要发现

• beta-DVCCA在Noisy MNIST上达到98.9%的测试准确率，优于标准DVCCA（93.7%）和基线联合模型（97.7%）。
• DVSIB达到93.9%的准确率，表明当对称压缩与数据的互信息结构匹配时性能得到提升。
• DVSIB和beta-DVCCA在误差与样本量的双对数图中表现出更陡峭的收敛斜率，表明在较少样本下学习速度更快。
• 该框架成功推导并实现了DVSIB，一种新方法，通过压缩两个变量以保留其表征间的互信息。
• 在DVSIB-private中，私有潜在变量保留了尺度和旋转因子，但未保留数字标签，证实了共享与私有因子的有效解耦。
• 当潜在维度被限制为2时，DVSIB和beta-DVCCA仍实现了有意义的聚类，表明对低维压缩具有鲁棒性。

Figure 3: Top: t-SNE plot of the latent space $Z_{Y}$ of DVSIB colored by the identity of digits. Top Right: Classification accuracy of an SVM trained on DVSIB’s $Z_{Y}$ latent space. The accuracy was evaluated for DVSIB with a parameter sweep of the trade-off parameter $\beta=2^{-5},...,2^{10}$ and