[Paper Review] FastText.zip: Compressing text classification models
This paper proposes FastText.zip, a method to compress text classification models using product quantization (PQ) with discriminative pruning and retraining, reducing model size by up to 1,000× with minimal accuracy loss. The approach achieves state-of-the-art memory-accuracy trade-offs, enabling sub-100kB models while maintaining high performance on standard benchmarks.
Online communities can be used to promote destructive behaviours, as in pro-Eating Disorder (ED) communities. Research needs annotated data to study these phenomena. Even though many platforms have already moderated this type of content, Twitter has not, and it can still be used for research purposes. In this paper, we unveiled emojis, words, and uncommon linguistic patterns within the ED Twitter community by using the Correlation Explanation (CorEx) algorithm on unstructured and non-annotated data to retrieve the topics. Then we annotated the dataset following these topics. We analysed then the use of CorEx and Word Mover’s Distance to retrieve automatically similar new sentences and augment the annotated dataset.
Motivation & Objective
- Address the challenge of deploying accurate text classifiers on memory-constrained devices such as smartphones.
- Reduce the memory footprint of text classification models—particularly word embeddings—without sacrificing classification accuracy.
- Explore compression techniques inspired by hashing and quantization to achieve extreme model compression while preserving performance.
- Provide a practical, reproducible baseline for efficient text classification models optimized for memory-accuracy trade-offs.
- Extend the fastText library with a new compression pipeline that enables compact, fast, and accurate models for real-world deployment.
Proposed method
- Apply optimized product quantization (OPQ) to compress word and output embeddings into compact codes, reducing memory usage by up to 1,000×.
- Use discriminative pruning based on embedding norm to remove low-impact features, improving compression efficiency and coverage.
- Introduce a two-stage retraining strategy: first quantize the input embeddings, then fine-tune the output classifier to recover accuracy.
- Employ magnitude/direction vector parametrization to enable efficient quantization of unit-norm vectors, compatible with PQ and LSH-based methods.
- Use hierarchical softmax and n-gram feature hashing to maintain model expressiveness while reducing dictionary size.
- Combine feature pruning, quantization, and retraining to minimize accuracy drop during extreme compression, especially at 64KiB and 32KiB model sizes.
Experimental results
Research questions
- RQ1Can product quantization be adapted to text classification models to achieve extreme compression without significant accuracy loss?
- RQ2How does discriminative pruning based on embedding norm compare to entropy-based pruning in terms of coverage and performance?
- RQ3What is the impact of quantizing both input and output weight matrices on model accuracy and memory usage?
- RQ4To what extent can model size be reduced (e.g., to under 64KiB) while maintaining acceptable classification accuracy?
- RQ5Can retraining after quantization effectively recover performance lost due to compression?
Key findings
- FastText.zip reduces model size by up to 1,000× compared to standard fastText, achieving models under 100kB on common benchmarks.
- At 64KiB, the model incurs only a 0.8% accuracy drop, and at 32KiB, the drop is 1.7%, demonstrating robustness to extreme compression.
- Quantizing both input and output matrices with OPQ maintains performance within 0.2% of the full model on the FlickrTag dataset.
- Norm-based pruning outperforms entropy-based pruning in coverage and accuracy, especially when combined with max-cover pruning heuristics.
- The method achieves state-of-the-art memory-accuracy trade-offs, outperforming both standard fastText and character-level CNNs in memory efficiency.
- Retraining after quantization significantly improves performance, especially when combined with pruning, enabling high-accuracy models at minimal size.
Read this paper
Open the full text in the Nubint reader and ask about any passage as you read.
Read this paper on NubintThis review was generated by AI from the original paper. Please check the paper before citing it.