[Paper Review] Correlational Neural Networks

Sarath Chandar, Mitesh M. Khapra+2 more|arXiv (Cornell University)|Apr 27, 2015

Natural Language Processing Techniques35 references4 citations

TL;DR

This paper proposes Correlational Neural Networks (CorrNet), a deep learning framework that jointly optimizes for reconstruction and correlation between views in a shared latent space. By explicitly maximizing correlation during training while enabling scalable, autoencoder-based reconstruction, CorrNet outperforms state-of-the-art methods in transfer learning, missing view reconstruction, and multilingual representation learning, particularly on cross-lingual tasks.

ABSTRACT

Common Representation Learning (CRL), wherein different descriptions (or views) of the data are embedded in a common subspace, is receiving a lot of attention recently. Two popular paradigms here are Canonical Correlation Analysis (CCA) based approaches and Autoencoder (AE) based approaches. CCA based approaches learn a joint representation by maximizing correlation of the views when projected to the common subspace. AE based methods learn a common representation by minimizing the error of reconstructing the two views. Each of these approaches has its own advantages and disadvantages. For example, while CCA based approaches outperform AE based approaches for the task of transfer learning, they are not as scalable as the latter. In this work we propose an AE based approach called Correlational Neural Network (CorrNet), that explicitly maximizes correlation among the views when projected to the common subspace. Through a series of experiments, we demonstrate that the proposed CorrNet is better than the above mentioned approaches with respect to its ability to learn correlated common representations. Further, we employ CorrNet for several cross language tasks and show that the representations learned using CorrNet perform better than the ones learned using other state of the art approaches.

Motivation & Objective

• To address the limitations of existing common representation learning methods, such as CCA’s poor scalability and autoencoders’ lack of explicit correlation learning.
• To develop a scalable, deep learning-based method that simultaneously enables reconstruction of views and maximizes correlation between their projections in a shared subspace.
• To leverage both parallel and non-parallel single-view data, which is common in real-world applications but underutilized by prior methods.
• To improve performance on downstream tasks such as cross-lingual document classification and transliteration equivalence detection using the learned representations.

Proposed method

• CorrNet uses a deep autoencoder architecture with shared latent layers for two views, X and Y, to learn a common k-dimensional representation.
• The model is trained using a combined loss function that includes reconstruction loss for both self-reconstruction and cross-reconstruction of views.
• A novel correlation regularization term is introduced to explicitly maximize the correlation between the projected representations of X and Y in the shared space.
• The training procedure is scalable and can incorporate additional single-view data, enhancing representation quality without requiring paired examples.
• The method uses a two-stream encoder and decoder setup, with shared weights in the bottleneck layer to enforce a common representation.
• Hyperparameters such as latent dimensionality and learning rate are tuned using validation data, with fixed dimensions (e.g., L=384) used in evaluation.

Experimental results

Research questions

• RQ1 Can a deep autoencoder-based method be designed to explicitly maximize correlation between views in a shared representation space while maintaining reconstruction capability?
• RQ2 How does CorrNet compare to CCA and other autoencoder-based methods in terms of scalability and performance on transfer learning and reconstruction tasks?
• RQ3 Can CorrNet effectively leverage non-parallel, single-view data to improve representation quality in low-resource settings?
• RQ4 Does the use of correlated common representations lead to better performance on cross-lingual tasks such as document classification and transliteration detection?
• RQ5 Can CorrNet outperform state-of-the-art methods like DCCAE in monolingual tasks such as bigram similarity prediction?

Key findings

• CorrNet achieved a Spearman’s correlation of 46.8 on the bigram similarity dataset, outperforming the previous SOTA method DCCAE (46.2) on average.
• On the cross-lingual document classification task, CorrNet achieved higher accuracy than CCA, DCCA, and DCCAE, demonstrating superior transfer learning performance.
• In transliteration equivalence detection, CorrNet outperformed all baseline methods, showing improved alignment of cross-lingual word forms.
• The model with 500 units and 384-dimensional latent space (CorrNet-500-384) yielded the best results across all evaluated tasks.
• CorrNet successfully reconstructed missing views with high fidelity, confirming its capability for multi-view data completion.
• The method demonstrated scalability and generalization benefits from single-view data, which other methods like CCA cannot utilize effectively.