[Paper Review] KATE: K-Competitive Autoencoder for Text

Yu Chen, Mohammed J. Zaki|arXiv (Cornell University)|May 4, 2017

Topic Modeling40 references19 citations

TL;DR

KATE is a k-competitive autoencoder that enhances text representation learning by enforcing competition among hidden neurons, where only the top-k neurons activate and redistribute energy from inactive neurons. It achieves state-of-the-art performance in document classification, regression, and retrieval, outperforming autoencoders, probabilistic topic models, and word embedding methods on multiple benchmarks.

ABSTRACT

Autoencoders have been successful in learning meaningful representations from image datasets. However, their performance on text datasets has not been widely studied. Traditional autoencoders tend to learn possibly trivial representations of text documents due to their confounding properties such as high-dimensionality, sparsity and power-law word distributions. In this paper, we propose a novel k-competitive autoencoder, called KATE, for text documents. Due to the competition between the neurons in the hidden layer, each neuron becomes specialized in recognizing specific data patterns, and overall the model can learn meaningful representations of textual data. A comprehensive set of experiments show that KATE can learn better representations than traditional autoencoders including denoising, contractive, variational, and k-sparse autoencoders. Our model also outperforms deep generative models, probabilistic topic models, and even word representation models (e.g., Word2Vec) in terms of several downstream tasks such as document classification, regression, and retrieval.

Motivation & Objective

• To address the limitations of traditional autoencoders in learning non-trivial, meaningful representations from high-dimensional, sparse, and power-law-distributed text data.
• To overcome the tendency of standard autoencoders to learn trivial or redundant representations due to data sparsity and high dimensionality.
• To develop a competitive learning mechanism in autoencoders that specializes neurons in recognizing distinct data patterns, improving semantic representation quality.
• To evaluate KATE against a broad range of models—including autoencoders, deep generative models, topic models, and word embeddings—across multiple downstream NLP tasks.
• To demonstrate that a shallow, single-hidden-layer architecture with competitive learning can surpass deeper or more complex models in text representation learning.

Proposed method

• KATE employs a k-competitive mechanism in the hidden layer, where only the k neurons with the highest activation values are selected as 'winners'.
• The model redistributes the aggregate activation potential of the non-winning (inactive) neurons to the k winning neurons via an energy amplification factor α.
• The energy redistribution is implemented through a modified activation function that incorporates the amplified potential from inactive neurons, enhancing feature specialization.
• The encoder uses a tanh activation function in the k-competitive layer, which empirically yields better performance than sigmoid.
• The model uses weight tying (W' = W^T) as a regularization technique to reduce overfitting and improve generalization.
• During inference, no competition is needed—the trained model directly encodes inputs using the fixed, specialized neuron weights.

Experimental results

Research questions

• RQ1 Can a k-competitive autoencoder learn more meaningful and semantically coherent representations of text documents than standard autoencoders?
• RQ2 How does neuron competition in the hidden layer affect the quality of learned document representations compared to non-competitive or sparse autoencoders?
• RQ3 Does KATE outperform deep generative models, probabilistic topic models, and word embedding methods in downstream NLP tasks like classification and retrieval?
• RQ4 What is the impact of hyperparameters such as k (number of winners), number of topics (neurons), and energy amplification α on model performance?
• RQ5 Can a shallow autoencoder with competitive learning achieve performance comparable to or better than deeper or more complex architectures in text representation learning?

Key findings

• KATE achieved 74.6% accuracy on the 20 Newsgroups dataset for document classification, significantly outperforming other autoencoders and competitive models.
• On document retrieval, KATE achieved a precision of 71.1% on the 20 Newsgroups dataset, outperforming all other models except VAE and DocNADE.
• The model's performance was highly sensitive to the choice of activation function, with tanh yielding 74.4% accuracy compared to 56.8% with sigmoid.
• Increasing the number of topics beyond 128 yielded only minor performance gains, indicating that 128 provides a good trade-off between complexity and performance.
• The energy amplification parameter α had a strong impact: increasing α from 0.0625 to 6.26 improved classification accuracy from 71.1% to 74.6%.
• KATE trained in 1,214 seconds on the 20 Newsgroups dataset, making it faster than deep generative models like DBN (15,281s) and DocNADE (4,787s), though slower than simpler autoencoders.