[Paper Review] Model compression for faster structural separation of macromolecules captured by Cellular Electron Cryo-Tomography

Jialiang Guo, Bo Zhou+3 more|arXiv (Cornell University)|Jan 31, 2018

Computational Physics and Python Applications8 references3 citations

TL;DR

This paper proposes knowledge distillation-based model compression to accelerate macromolecule classification in cellular electron cryo-tomography (CECT). By training compact student networks on soft labels from a deep teacher network (DSRF3D-v2), the method reduces model size by up to 113× and inference time by 2.85×, while preserving 93.8% of the original accuracy, enabling efficient deployment on large-scale CECT datasets.

ABSTRACT

Electron Cryo-Tomography (ECT) enables 3D visualization of macromolecule structure inside single cells. Macromolecule classification approaches based on convolutional neural networks (CNN) were developed to separate millions of macromolecules captured from ECT systematically. However, given the fast accumulation of ECT data, it will soon become necessary to use CNN models to efficiently and accurately separate substantially more macromolecules at the prediction stage, which requires additional computational costs. To speed up the prediction, we compress classification models into compact neural networks with little in accuracy for deployment. Specifically, we propose to perform model compression through knowledge distillation. Firstly, a complex teacher network is trained to generate soft labels with better classification feasibility followed by training of customized student networks with simple architectures using the soft label to compress model complexity. Our tests demonstrate that our compressed models significantly reduce the number of parameters and time cost while maintaining similar classification accuracy.

Motivation & Objective

• Address the growing computational bottleneck in processing millions of subtomograms from high-throughput CECT data.
• Reduce the high inference cost of large, accurate 3D-CNN models used for macromolecular classification.
• Enable efficient deployment of deep learning models on large-scale CECT datasets with limited computational resources.
• Maintain high classification accuracy despite significant model compression for real-world biological applications.

Proposed method

• Train a complex DSRF3D-v2 teacher network on simulated CECT subtomograms to generate high-quality soft labels.
• Design three progressively smaller student networks (DSRF3D-v2-s1, s2, s3) with simplified architectures.
• Train student networks using knowledge distillation, where soft labels from the teacher network guide learning.
• Use a temperature-scaled softmax function (T=5) to enhance knowledge transfer from teacher to student.
• Optimize student models with stochastic gradient descent and categorical cross-entropy loss with Nesterov momentum.
• Apply early stopping based on validation loss to prevent overfitting and ensure generalization.

Experimental results

Research questions

• RQ1 Can knowledge distillation effectively compress large 3D-CNN models for CECT subtomogram classification without significant accuracy loss?
• RQ2 What is the trade-off between model compression rate, inference speed, and classification accuracy in 3D biological image analysis?
• RQ3 How does knowledge distillation from a teacher network improve the performance of smaller student networks compared to training on hard labels alone?
• RQ4 Which student network architecture achieves the best balance between compression, speed, and accuracy in this context?
• RQ5 To what extent can model compression enable practical deployment of deep learning models on massive CECT datasets?

Key findings

• The DSRF3D-v2-s1 student model achieved a 18.1× compression rate, reducing parameters from 18.3M to 1.0M while maintaining 93.8% of the teacher model’s accuracy.
• The DSRF3D-v2-s1 model reduced inference time from 15.91 seconds to 9.41 seconds per million subtomograms, achieving a 1.69× speedup.
• Training student networks on soft labels from the teacher model improved accuracy by up to 8.16 percentage points compared to training on hard labels alone.
• The DSRF3D-v2-s3 model achieved the highest compression rate (113.3×) but suffered the largest accuracy drop, retaining only 89.83% of the teacher’s accuracy.
• Knowledge distillation enabled student models to generalize better than those trained on hard labels, demonstrating the value of soft label guidance.
• The results show that model compression via distillation is highly effective for 3D-CNNs in CECT, enabling efficient processing of large-scale biological datasets.