[Paper Review] C-Mixup: Improving Generalization in Regression

Huaxiu Yao, Yiping Wang+3 more|arXiv (Cornell University)|Oct 11, 2022

Domain Adaptation and Few-Shot Learning13 citations

TL;DR

C-Mixup is a novel data augmentation method for regression that improves generalization by sampling training pairs based on label similarity rather than uniform or feature-based sampling. By using a Gaussian kernel over label distances, it reduces noisy interpolations, achieves state-of-the-art performance in in-distribution generalization, task generalization, and out-of-distribution robustness across 11 datasets, with improvements of up to 6.56% over prior methods.

ABSTRACT

Improving the generalization of deep networks is an important open challenge, particularly in domains without plentiful data. The mixup algorithm improves generalization by linearly interpolating a pair of examples and their corresponding labels. These interpolated examples augment the original training set. Mixup has shown promising results in various classification tasks, but systematic analysis of mixup in regression remains underexplored. Using mixup directly on regression labels can result in arbitrarily incorrect labels. In this paper, we propose a simple yet powerful algorithm, C-Mixup, to improve generalization on regression tasks. In contrast with vanilla mixup, which picks training examples for mixing with uniform probability, C-Mixup adjusts the sampling probability based on the similarity of the labels. Our theoretical analysis confirms that C-Mixup with label similarity obtains a smaller mean square error in supervised regression and meta-regression than vanilla mixup and using feature similarity. Another benefit of C-Mixup is that it can improve out-of-distribution robustness, where the test distribution is different from the training distribution. By selectively interpolating examples with similar labels, it mitigates the effects of domain-associated information and yields domain-invariant representations. We evaluate C-Mixup on eleven datasets, ranging from tabular to video data. Compared to the best prior approach, C-Mixup achieves 6.56%, 4.76%, 5.82% improvements in in-distribution generalization, task generalization, and out-of-distribution robustness, respectively. Code is released at https://github.com/huaxiuyao/C-Mixup.

Motivation & Objective

• To address the poor generalization of vanilla mixup in regression due to arbitrary label interpolation from dissimilar examples.
• To improve in-distribution generalization, task generalization in meta-learning, and out-of-distribution robustness in regression tasks.
• To reduce computational cost and improve interpolation quality by replacing feature-based similarity with label-based similarity for pair sampling.
• To develop a scalable, robust, and theoretically grounded method for regression that mitigates spurious correlations and domain shift.

Proposed method

• C-Mixup uses a Gaussian kernel to compute sampling probabilities for mixing pairs based on the Euclidean distance between their continuous labels.
• It replaces uniform or feature-based sampling with label similarity, ensuring that only examples with close labels are likely to be mixed.
• The method integrates into standard training by linearly interpolating features and labels using a beta-distributed mixing coefficient λ ~ Beta(α, α).
• It extends to meta-learning via MetaMix, where C-Mixup improves task generalization in MAML-based few-shot regression.
• The algorithm is applied in a two-stage process: first computing hidden representations for distance calculation, then performing mixup using label or representation-based similarity.
• It uses a bandwidth parameter σ in the Gaussian kernel to control the sensitivity of similarity weighting, with empirical robustness across a wide range of σ values.

Experimental results

Research questions

• RQ1 Can label-based sampling in mixup improve generalization in regression tasks compared to uniform or feature-based sampling?
• RQ2 Does C-Mixup enhance out-of-distribution robustness by encouraging domain-invariant representations?
• RQ3 How does C-Mixup perform in meta-learning settings for few-shot regression compared to vanilla mixup?
• RQ4 Is C-Mixup robust to hyperparameter choices such as the bandwidth σ and shape parameter α of the Beta distribution?
• RQ5 How does C-Mixup perform under label noise, and does it maintain advantages over standard mixup?

Key findings

• C-Mixup achieves a 6.56% improvement in in-distribution generalization over the best prior method on the evaluated regression tasks.
• It improves task generalization by 4.76% in meta-learning settings compared to vanilla MetaMix.
• C-Mixup achieves a 5.82% improvement in out-of-distribution robustness, particularly under covariate shift.
• The method is robust to hyperparameter choices: performance remains stable across a wide range of bandwidth σ and shape parameter α.
• C-Mixup outperforms both vanilla mixup and feature-similarity-based mixup in all 11 datasets, including tabular, image, and video data.
• In the presence of 30% label noise, C-Mixup maintains superior performance, reducing RMSE by 1.0% on average compared to ERM and 1.3% compared to mixup.