[Paper Review] Concave losses for robust dictionary learning

Rafael Will M de Araujo, Roberto Hirata+1 more|arXiv (Cornell University)|Nov 2, 2017

Sparse and Compressive Sensing Techniques8 references3 citations

TL;DR

This paper proposes a generic framework for robust dictionary learning using concave loss functions to reduce sensitivity to outliers. By composing a concave function with the square root of reconstruction error, the method enables effective outlier detection and improved dictionary quality, outperforming K-SVD and LC-KSVD on synthetic and real-world data, including face classification on MORPH-II.

ABSTRACT

Traditional dictionary learning methods are based on quadratic convex loss function and thus are sensitive to outliers. In this paper, we propose a generic framework for robust dictionary learning based on concave losses. We provide results on composition of concave functions, notably regarding super-gradient computations, that are key for developing generic dictionary learning algorithms applicable to smooth and non-smooth losses. In order to improve identification of outliers, we introduce an initialization heuristic based on undercomplete dictionary learning. Experimental results using synthetic and real data demonstrate that our method is able to better detect outliers, is capable of generating better dictionaries, outperforming state-of-the-art methods such as K-SVD and LC-KSVD.

Motivation & Objective

• To address the sensitivity of traditional dictionary learning to outliers due to quadratic convex loss functions.
• To develop a generic algorithm for robust dictionary learning using concave loss functions that downweight large reconstruction errors.
• To improve outlier identification through a novel undercomplete dictionary initialization heuristic.
• To demonstrate superior performance on both synthetic and real-world datasets, including face attribute classification.
• To provide a unified framework applicable to smooth and non-smooth concave losses via majorization-minimization.

Proposed method

• Formulates dictionary learning using a composite concave loss function F(u) = g(√u), where g is a non-decreasing concave function, to downweight large reconstruction errors.
• Employs a majorization-minimization algorithm based on supergradient computation to solve the non-convex optimization problem efficiently.
• Introduces an undercomplete dictionary initialization strategy to enhance outlier detection during training.
• Applies sparsity-inducing ℓ₁ penalty on coefficients and unit-norm constraints on dictionary atoms.
• Uses superdifferential-based upper bounds for concave functions to iteratively minimize the objective.
• Employs the SIFT descriptor and BoVW with SVM for evaluation on real datasets like MORPH-II.

Experimental results

Research questions

• RQ1 Can a generic framework based on concave loss functions improve robustness in dictionary learning compared to standard quadratic loss?
• RQ2 How does the composition of concave functions (e.g., g(√·)) enhance outlier detection in dictionary learning?
• RQ3 Does undercomplete dictionary initialization improve the identification of outliers during training?
• RQ4 How does the proposed method compare to state-of-the-art methods like K-SVD and LC-KSVD in terms of dictionary quality and classification accuracy?
• RQ5 To what extent is the method stable under varying outlier ratios and data dimensions?

Key findings

• The proposed method detected 47 out of 50 true outliers in a synthetic experiment using the log function, significantly outperforming K-SVD, which detected none.
• With the log function, the method achieved an AUROC of 0.98 on synthetic data with 32-dimensional features, showing high stability across sample sizes and outlier ratios.
• On the MORPH-II dataset, the proposed method achieved 96.90% accuracy for ethnicity classification and 85.79% for gender classification, surpassing K-SVD (96.23% and 81.88%) and LC-KSVD variants.
• The undercomplete initialization strategy improved performance, especially when the number of atoms exceeded the data dimension, and maintained high AUROC even with increasing outlier proportions.
• The method remained stable and effective across varying numbers of samples and outlier ratios, demonstrating robustness in real-world settings.
• The use of concave functions like log(ε + u) and capped-ℓ₁ provided better outlier detection than linear or convex alternatives.