[论文解读] Variational Deep Semantic Hashing for Text Documents

Suthee Chaidaroon, Yi Fang|arXiv (Cornell University)|Aug 11, 2017

Advanced Image and Video Retrieval Techniques参考文献 24被引用 6

一句话总结

本文提出变分深度语义哈希（VDSH），一种基于变分推断与重参数化技巧的深度生成模型，用于学习文本文档的紧凑二进制编码。通过将文档建模为由共享语义因子和文档特定的私有变量生成，该方法在语义哈希任务中实现了最先进性能，尤其在有标签指导的监督设置下表现突出。

ABSTRACT

As the amount of textual data has been rapidly increasing over the past decade, efficient similarity search methods have become a crucial component of large-scale information retrieval systems. A popular strategy is to represent original data samples by compact binary codes through hashing. A spectrum of machine learning methods have been utilized, but they often lack expressiveness and flexibility in modeling to learn effective representations. The recent advances of deep learning in a wide range of applications has demonstrated its capability to learn robust and powerful feature representations for complex data. Especially, deep generative models naturally combine the expressiveness of probabilistic generative models with the high capacity of deep neural networks, which is very suitable for text modeling. However, little work has leveraged the recent progress in deep learning for text hashing. In this paper, we propose a series of novel deep document generative models for text hashing. The first proposed model is unsupervised while the second one is supervised by utilizing document labels/tags for hashing. The third model further considers document-specific factors that affect the generation of words. The probabilistic generative formulation of the proposed models provides a principled framework for model extension, uncertainty estimation, simulation, and interpretability. Based on variational inference and reparameterization, the proposed models can be interpreted as encoder-decoder deep neural networks and thus they are capable of learning complex nonlinear distributed representations of the original documents. We conduct a comprehensive set of experiments on four public testbeds. The experimental results have demonstrated the effectiveness of the proposed supervised learning models for text hashing.

研究动机与目标

• 为解决传统哈希方法在学习文本文档表达性与灵活性表征方面的局限性。
• 将深度生成模型与变分推断相结合，实现可扩展且可解释的文本哈希。
• 通过引入文档标签与私有潜在变量，提升语义哈希性能，降低噪声并改善表征质量。
• 为文本哈希提供一个支持不确定性估计、模型扩展与可解释性的系统性框架。

提出的方法

• 提出一种基于变分自编码器（VAEs）的深度生成模型，其中文档词项由共享语义因子与文档特定的私有因子生成。
• 使用带重参数化的变分推断，实现通过反向传播进行端到端训练。
• 建模文档、标签与潜在变量的联合概率，将其分解为具有高斯先验的潜在因子生成过程。
• 采用蒙特卡洛采样近似目标函数中不可计算的期望，支持随机优化。
• 推导出一个可微的下界（ELBO），包含重构损失、标签预测损失以及语义与私有潜在变量的KL散度项。
• 应用重参数化技巧，将潜在变量采样表示为 $\bm{s}^{(m)} = \bm{\epsilon}_s^{(m)} \odot \bm{\sigma}_s + \bm{\mu}_s$，实现通过随机节点的反向传播。

实验结果

研究问题

• RQ1 基于变分推断的深度生成模型是否能在学习紧凑且高效的文本文档二进制编码方面超越传统哈希方法？
• RQ2 引入文档标签在多大程度上提升了所学语义哈希的质量？
• RQ3 建模文档特定的私有因子在多大程度上增强了表征鲁棒性并降低了文本哈希中的噪声？
• RQ4 该概率生成框架是否能够支持不确定性估计、可解释性与模型扩展？

主要发现

• VDSH的监督变体在四个公开文本哈希基准上达到最先进性能，平均平均精度（mAP）显著优于现有方法。
• 引入文档标签显著提升了mAP，证明了监督信号在学习有意义语义编码中的有效性。
• 引入私有潜在变量的模型性能优于不引入该变量的模型，表明建模文档特定因子有助于降低噪声并提升泛化能力。
• 使用变分推断支持端到端训练，并为不确定性量化与模型可解释性提供了系统性框架。
• 目标函数同时包含重构项与标签预测项，KL散度正则化确保学习到的潜在分布保持接近先验分布。