[Paper Review] The role of dimensionality reduction in linear classification

Weiran Wang, Miguel Á. Carreira-Perpiñán|arXiv (Cornell University)|May 26, 2014

Neural Networks and Applications27 references3 citations

TL;DR

This paper proposes a novel, efficient algorithm for jointly optimizing a nonlinear dimensionality reduction (DR) mapping and a linear classifier using auxiliary coordinates, enabling end-to-end training that outperforms traditional filter methods. The method achieves state-of-the-art classification accuracy with fast inference by collapsing classes in latent space for maximum linear separability, while allowing users to trade off runtime and accuracy via the number of basis functions.

ABSTRACT

Dimensionality reduction (DR) is often used as a preprocessing step in classification, but usually one first fixes the DR mapping, possibly using label information, and then learns a classifier (a filter approach). Best performance would be obtained by optimizing the classification error jointly over DR mapping and classifier (a wrapper approach), but this is a difficult nonconvex problem, particularly with nonlinear DR. Using the method of auxiliary coordinates, we give a simple, efficient algorithm to train a combination of nonlinear DR and a classifier, and apply it to a RBF mapping with a linear SVM. This alternates steps where we train the RBF mapping and a linear SVM as usual regression and classification, respectively, with a closed-form step that coordinates both. The resulting nonlinear low-dimensional classifier achieves classification errors competitive with the state-of-the-art but is fast at training and testing, and allows the user to trade off runtime for classification accuracy easily. We then study the role of nonlinear DR in linear classification, and the interplay between the DR mapping, the number of latent dimensions and the number of classes. When trained jointly, the DR mapping takes an extreme role in eliminating variation: it tends to collapse classes in latent space, erasing all manifold structure, and lay out class centroids so they are linearly separable with maximum margin.

Motivation & Objective

• To address the limitation of filter-based dimensionality reduction, where DR and classifier are trained sequentially, leading to suboptimal performance.
• To develop a scalable, efficient, and generic wrapper approach that jointly optimizes the DR mapping and classifier to minimize classification error directly.
• To investigate the true role of nonlinear DR in linear classification, particularly how the latent space representation affects generalization and separability.
• To provide a practical, plug-and-play method that reuses standard regression and SVM solvers, enabling fast training and testing with tunable accuracy/runtime trade-offs.
• To demonstrate that optimal DR under joint optimization erases all manifold structure and collapses classes to maximize margin in latent space.

Proposed method

• Uses the method of auxiliary coordinates to transform the nonconvex joint optimization of DR mapping F and linear SVM classifier g into a sequence of convex subproblems.
• Alternates between training the RBF-based nonlinear DR mapping F via regression and updating the linear SVM classifier g via standard SVM optimization.
• Incorporates a closed-form coordination step between F and g that ensures the joint objective is minimized at each iteration.
• Employs a regularization term λR(F) on the DR mapping to prevent overfitting and control model complexity.
• Applies the algorithm to a RBF network for F and a linear SVM for g, enabling explicit, fast inference on test samples.
• The approach is scalable and parallelizable, reusing standard implementations of regression and SVM solvers.

Experimental results

Research questions

• RQ1 What is the true role of nonlinear dimensionality reduction in improving linear classification performance when DR and classifier are jointly optimized?
• RQ2 How does the latent space representation change when the classification error is minimized jointly over DR and classifier, rather than using proxy objectives?
• RQ3 Can a joint optimization framework outperform standard filter-based DR methods like LDA or PCA in terms of classification accuracy and efficiency?
• RQ4 To what extent does the DR mapping preserve or erase data manifold structure under joint optimization with a linear classifier?
• RQ5 How does the number of latent dimensions L and the number of classes influence the optimal DR mapping and classification performance?

Key findings

• The proposed joint optimization algorithm achieves state-of-the-art classification error on benchmark datasets while maintaining fast training and inference times.
• When jointly optimized, the nonlinear DR mapping collapses all data points of the same class into a single point in the latent space, erasing all manifold structure.
• The optimal DR mapping places class centroids in the latent space such that they are maximally linearly separable with maximum margin.
• The method effectively eliminates both independent random noise and label-invariant variations (unwanted degrees of freedom), achieving a form of radical denoising.
• The algorithm converges to a local minimum of the joint objective and can be seen as an iterated filter approach with provable convergence and improved performance.
• The results suggest that traditional filter methods based on class separability or intra-class scatter are justified, but can be surpassed by direct joint optimization.