[Paper Review] Variational Interaction Information Maximization for Cross-domain Disentanglement

HyeongJoo Hwang, Geon-Hyeong Kim+2 more|arXiv (Cornell University)|Dec 8, 2020

Domain Adaptation and Few-Shot Learning43 references21 citations

TL;DR

This paper proposes Interaction Information Auto-Encoder (IIAE), a variational inference-based generative model that disentangles domain-invariant and domain-specific representations using information-theoretic regularization, without adversarial training or gradient reversal. IIAE achieves state-of-the-art performance in zero-shot sketch-based image retrieval (mAP: 0.573, P@100: 0.659) on Sketchy Extended, even without external knowledge like word embeddings.

ABSTRACT

Cross-domain disentanglement is the problem of learning representations partitioned into domain-invariant and domain-specific representations, which is a key to successful domain transfer or measuring semantic distance between two domains. Grounded in information theory, we cast the simultaneous learning of domain-invariant and domain-specific representations as a joint objective of multiple information constraints, which does not require adversarial training or gradient reversal layers. We derive a tractable bound of the objective and propose a generative model named Interaction Information Auto-Encoder (IIAE). Our approach reveals insights on the desirable representation for cross-domain disentanglement and its connection to Variational Auto-Encoder (VAE). We demonstrate the validity of our model in the image-to-image translation and the cross-domain retrieval tasks. We further show that our model achieves the state-of-the-art performance in the zero-shot sketch based image retrieval task, even without external knowledge. Our implementation is publicly available at: https://github.com/gr8joo/IIAE

Motivation & Objective

• To address cross-domain disentanglement by learning representations that are both disentangled and informative, without relying on heuristic regularization techniques.
• To formalize the joint learning of domain-invariant and domain-specific representations as a constrained optimization problem grounded in information theory.
• To derive a tractable lower bound of the objective that integrates naturally with the VAE's Evidence Lower Bound (ELBO).
• To demonstrate the effectiveness of the proposed method in image-to-image translation, cross-domain retrieval, and zero-shot sketch-based image retrieval.
• To show that IIAE outperforms state-of-the-art models in ZS-SBIR even without external knowledge such as word embeddings or attribute annotations.

Proposed method

• The method formulates cross-domain disentanglement as a joint optimization of multiple information constraints: maximizing mutual information between shared representation and input while minimizing mutual information between shared representation and domain-specific factors.
• It introduces a variational inference framework using a structured posterior approximation qφ(zx, zs, zy|x, y) = qφX(zx|x)qφS(zs|x, y)qφY(zy|y), enabling end-to-end training.
• The objective is derived as a lower bound that combines the ELBO with two information regularization terms: one enforcing the shared representation ZS to be a minimal sufficient statistic (via minimizing I(ZS; ZX, ZY)), and another enforcing ZS to be maximally informative about (X, Y) (via maximizing I(ZS; X, Y)).
• The model uses a hyperparameter λ to balance the trade-off between invariance and compressiveness in the shared representation.
• The framework is implemented as a generative model trained via backpropagation, with no need for adversarial training or gradient reversal layers.
• The shared representation ZS is used for downstream tasks such as image retrieval via cosine similarity.

Experimental results

Research questions

• RQ1 Can a principled information-theoretic framework be designed to jointly learn disentangled domain-invariant and domain-specific representations without adversarial training or gradient reversal?
• RQ2 How can the mutual information constraints between shared and exclusive representations be approximated in a tractable way for end-to-end training?
• RQ3 To what extent does the proposed method improve disentanglement quality and downstream performance in cross-domain image translation and retrieval tasks?
• RQ4 Can the model achieve state-of-the-art performance in zero-shot sketch-based image retrieval without relying on external knowledge like word embeddings or attribute labels?
• RQ5 What is the role of the shared representation as a minimal sufficient statistic in generalization to unseen classes?

Key findings

• IIAE achieves state-of-the-art performance on the zero-shot sketch-based image retrieval (ZS-SBIR) benchmark using the Sketchy Extended dataset, with mAP of 0.573 and P@100 of 0.659, outperforming models that use external knowledge such as word embeddings or WordNet.
• The model outperforms specialized baselines like LCALE (mAP: 0.476) and SEM-PCYC (mAP: 0.349), demonstrating its effectiveness even without attribute or word-level supervision.
• Ablation studies confirm that both information regularization terms—minimizing I(ZS; ZX, ZY) and maximizing I(ZS; X, Y)—are essential for performance, with λ controlling the balance between invariance and informativeness.
• Qualitative results show that even incorrectly retrieved images often share visual or semantic similarities with the query, indicating that the shared representation captures meaningful semantic structure.
• The method generalizes well to unseen classes, as evidenced by high performance on zero-shot retrieval, which is attributed to the shared representation being a minimal sufficient statistic.
• The model achieves strong results in image-to-image translation and cross-domain retrieval, validating its broader applicability beyond ZS-SBIR.