[Paper Review] Learning Less-Overlapping Representations

Pengtao Xie, Hongbao Zhang+1 more|arXiv (Cornell University)|Nov 25, 2017

Domain Adaptation and Few-Shot Learning38 references3 citations

TL;DR

This paper proposes a novel regularization method, LDD-L1, that encourages less-overlapping supports in representation learning by simultaneously promoting sparsity and near-orthogonality of weight vectors. Applied to neural networks and sparse coding, the method improves model interpretability and generalization, achieving state-of-the-art performance on PTB language modeling and CIFAR-10 image classification with lower perplexity and error rates compared to L1 and orthogonality-only regularizers.

ABSTRACT

In representation learning (RL), how to make the learned representations easy to interpret and less overfitted to training data are two important but challenging issues. To address these problems, we study a new type of regulariza- tion approach that encourages the supports of weight vectors in RL models to have small overlap, by simultaneously promoting near-orthogonality among vectors and sparsity of each vector. We apply the proposed regularizer to two models: neural networks (NNs) and sparse coding (SC), and develop an efficient ADMM-based algorithm for regu- larized SC. Experiments on various datasets demonstrate that weight vectors learned under our regularizer are more interpretable and have better generalization performance.

Motivation & Objective

• To improve interpretability of representation learning models by reducing redundancy among learned weight vectors.
• To reduce overfitting and improve generalization by constraining the structural complexity of learned representations.
• To develop a unified regularization approach that jointly promotes sparsity and near-orthogonality of weight vectors.
• To demonstrate the effectiveness of the proposed regularizer across diverse models and datasets, including NLP and image classification tasks.

Proposed method

• Proposes a smooth regularizer, LDD-L1, that encourages weight vectors to be sparse and close to orthogonal, thereby minimizing overlap in their supports.
• Defines a non-overlapness score based on the Jaccard index of vector supports to measure inter-vector overlap.
• Uses L1 regularization to promote sparsity in individual weight vectors and orthogonalization constraints to reduce support overlap.
• Develops an efficient ADMM-based optimization algorithm for solving the regularized sparse coding problem.
• Applies the LDD-L1 regularizer to both deep neural networks and sparse coding models, enabling end-to-end training with improved generalization.

Experimental results

Research questions

• RQ1 Can reducing overlap in the supports of learned weight vectors improve model interpretability in representation learning?
• RQ2 Does enforcing sparsity and near-orthogonality in weight vectors lead to better generalization on unseen data?
• RQ3 How does the proposed LDD-L1 regularizer compare to existing regularizers like L1 and orthogonality-promoting methods in terms of performance?
• RQ4 Can the LDD-L1 regularizer be effectively applied to both neural networks and sparse coding models?

Key findings

• On the PTB language modeling dataset, the LDD-L1 regularizer reduced test perplexity to 71.1, outperforming the unregularized baseline (72.3) and other regularizers.
• On CIFAR-10, the LDD-L1 regularizer achieved a test error rate of 3.60%, surpassing the unregularized WideResNet (3.89%) and other regularizers like L1 (3.81%) and orthogonality-only methods (e.g., AC-WideResNet at 3.63%).
• The LDD-L1 regularizer achieved lower error rates than both sparsity-only (L1) and orthogonality-only regularizers, demonstrating the superiority of jointly promoting sparsity and orthogonality.
• The proposed ADMM-based algorithm for regularized sparse coding converges efficiently and enables effective learning of less-overlapping representations.