[Paper Review] Efficient Domain Generalization via Common-Specific Low-Rank Decomposition

Vihari Piratla, Praneeth Netrapalli+1 more|arXiv (Cornell University)|Mar 28, 2020

Domain Adaptation and Few-Shot Learning21 references57 citations

TL;DR

Introduces CSD, a common-specific low-rank decomposition of the final layer for domain generalization, with an orthogonality constraint and a dedicated common loss, achieving competitive accuracy and faster training than meta-learning baselines across image and speech domains.

ABSTRACT

Domain generalization refers to the task of training a model which generalizes to new domains that are not seen during training. We present CSD (Common Specific Decomposition), for this setting,which jointly learns a common component (which generalizes to new domains) and a domain specific component (which overfits on training domains). The domain specific components are discarded after training and only the common component is retained. The algorithm is extremely simple and involves only modifying the final linear classification layer of any given neural network architecture. We present a principled analysis to understand existing approaches, provide identifiability results of CSD,and study effect of low-rank on domain generalization. We show that CSD either matches or beats state of the art approaches for domain generalization based on domain erasure, domain perturbed data augmentation, and meta-learning. Further diagnostics on rotated MNIST, where domains are interpretable, confirm the hypothesis that CSD successfully disentangles common and domain specific components and hence leads to better domain generalization.

Motivation & Objective

• Motivate and analyze domain generalization (DG) in multi-domain settings to prevent overfitting to training domains.
• Propose a principled decomposition-based DG method (CSD) that separates a domain-general common component from domain-specific low-rank components.
• Establish identifiability of the shared component and study how the low-rank choice affects domain generalization.
• Show that CSD is faster and often more accurate than gradient-based meta-learning and other DG methods across diverse datasets.
• Extend the approach to neural networks by constraining the decomposition to the final layer and introducing a common loss plus orthogonality constraints.

Proposed method

• Decompose the final linear classification layer into a common component and a low-rank domain-specific part with an orthogonality constraint between them.
• Train domain-specific softmax parameters w_i as w_c + W_s γ_i, and impose a loss on both w_i and w_c to encourage learning of shared features.
• Use an orthonormal regularizer to promote w_c ⟂ span(W_s) and control parameter norms.
• Provide an algorithm (CSD) that updates θ (feature extractor), w_c, W_s, and γ_i jointly, including a common loss term and a specialized loss term, plus the orthonormality regularizer.
• Derive identifiability results: w_c corresponds to e_c projected orthogonally to domain-specific components, ensuring a well-defined shared classifier.
• Analyze the role of the rank k in the low-rank decomposition, with a principled Eckart–Young–Mirsky-like approach to compute the decomposition and discuss trade-offs between bias and noise.

Experimental results

Research questions

• RQ1 Can the shared domain-general classifier w_c be uniquely identified within a common-specific decomposition framework?
• RQ2 How does the rank k of the domain-specific component affect domain generalization performance and noise suppression?
• RQ3 Does restricting the decomposition to the final layer with orthogonality and a common loss yield competitive or superior domain generalization compared to meta-learning and data-augmentation DG methods?
• RQ4 Is CSD scalable and faster than gradient-based DG methods while maintaining or improving accuracy across many training domains?
• RQ5 Do experiments on image and speech tasks support the hypothesis that CSD disentangles common and domain-specific components to improve out-of-domain generalization?

Key findings

• CSD matches or surpasses state-of-the-art DG methods based on domain erasure, augmentation, and meta-learning across multiple datasets.
• CSD offers an order-of-magnitude speedup over representative meta-learning DG methods while delivering higher accuracy when many domains are available.
• On rotated MNIST and other interpretable domains, CSD demonstrates clear disentanglement of shared versus domain-specific components.
• Ablation studies show the combination of orthogonality, common loss, and domain-specific loss is critical for gains beyond baseline ERM.
• The optimal rank k depends on the number of domains and data characteristics, with k=1 often favorable for moderate domain counts and higher k benefiting very large domain sets; cross-validation provides robust performance.
• Across image and speech tasks, CSD consistently improves out-of-domain accuracy with modest training-time overhead compared to ERM.