[Paper Review] k-Sparse Autoencoders

Alireza Makhzani, Brendan J. Frey|arXiv (Cornell University)|Dec 19, 2013

Domain Adaptation and Few-Shot LearningComputer Science14 references148 citations

TL;DR

This paper introduces k-sparse autoencoders, a fast and efficient method for learning sparse representations by retaining only the k largest hidden unit activations in a linear autoencoder. The approach achieves state-of-the-art classification performance on MNIST and NORB, outperforming denoising autoencoders, dropout, RBMs, and standard sparse autoencoders, with minimal computational overhead due to its simple, differentiable sparsity mechanism.

ABSTRACT

Recently, it has been observed that when representations are learnt in a way that encourages sparsity, improved performance is obtained on classification tasks. These methods involve combinations of activation functions, sampling steps and different kinds of penalties. To investigate the effectiveness of sparsity by itself, we propose the k-sparse autoencoder, which is an autoencoder with linear activation function, where in hidden layers only the k highest activities are kept. When applied to the MNIST and NORB datasets, we find that this method achieves better classification results than denoising autoencoders, networks trained with dropout, and RBMs. k-sparse autoencoders are simple to train and the encoding stage is very fast, making them well-suited to large problem sizes, where conventional sparse coding algorithms cannot be applied.

Motivation & Objective

• To investigate whether sparsity alone, without additional nonlinearities or regularization, can yield superior representation learning.
• To develop a computationally efficient alternative to conventional sparse coding, which is slow due to iterative optimization in dictionary learning and sparse coding steps.
• To evaluate the effectiveness of k-sparse autoencoders as a pretraining method for supervised deep learning on standard vision benchmarks.
• To compare the performance of k-sparse autoencoders against established methods such as denoising autoencoders, dropout, and RBMs in both shallow and deep learning setups.

Proposed method

• The k-sparse autoencoder uses a linear autoencoder with tied weights and linear activation functions.
• After computing the hidden representation z = W^T x + b, only the k largest activations are retained, and all others are set to zero, enforcing exact sparsity.
• The sparsity is implemented via a hard thresholding operation: z_sparse = z * one_hot(top_k(z)), where top_k identifies the indices of the k largest values.
• The method is trained end-to-end using standard backpropagation, with the sparsity constraint enforced during the forward pass.
• For downstream classification, the k-sparse representation is used as input to a classifier, and during fine-tuning, the same k-largest selection is applied to maintain consistency.
• The approach supports both shallow and deep architectures via greedy layer-wise pre-training, followed by discriminative fine-tuning.

Experimental results

Research questions

• RQ1 Can exact sparsity in hidden representations, enforced via a fixed k-largest selection, lead to better representation learning than traditional sparse coding or other regularization techniques?
• RQ2 How does the choice of k affect the quality of learned representations and downstream classification accuracy?
• RQ3 Can k-sparse autoencoders serve as an effective pretraining method for deep neural networks, comparable to RBMs or denoising autoencoders?
• RQ4 Does the absence of nonlinear activation functions and additional penalties improve performance when sparsity is the sole regularizer?

Key findings

• On the MNIST dataset, the k-sparse autoencoder achieved a test error of 0.97% when used in a deep architecture with layer-wise pre-training, outperforming all baselines including denoising autoencoders and dropout.
• For shallow networks on MNIST, the k-sparse autoencoder achieved 1.08% error with k=25, surpassing denoising autoencoders (1.20%) and dropout (1.05%) when fine-tuned.
• On the NORB dataset, the deep k-sparse autoencoder achieved 7.4% error with k=150, outperforming the next best method (third-order RBM at 6.5%) in the shallow setting.
• The k-sparse autoencoder demonstrated superior performance compared to RBMs, denoising autoencoders, and dropout-based autoencoders in both shallow and deep supervised learning setups.
• The method is computationally efficient: encoding is fast because it only requires identifying the top k activations, avoiding expensive iterative solvers used in traditional sparse coding.
• The results show that sparsity alone—without additional nonlinearities or penalties—can yield state-of-the-art performance, highlighting the power of structured sparsity in representation learning.