[Paper Review] Decomposing Convolutional Neural Networks into Reusable and Replaceable Modules

Rangeet Pan, Hridesh Rajan|arXiv (Cornell University)|Oct 11, 2021

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes decomposing Convolutional Neural Networks (CNNs) into class-specific modules—each acting as a binary classifier for a single output class—enabling reuse and replacement without retraining. The method incurs only a 1.77% top-1 accuracy drop and reduces CO₂e emissions by up to 37× compared to training from scratch, enabling efficient, sustainable model adaptation.

ABSTRACT

Training from scratch is the most common way to build a Convolutional Neural Network (CNN) based model. What if we can build new CNN models by reusing parts from previously build CNN models? What if we can improve a CNN model by replacing (possibly faulty) parts with other parts? In both cases, instead of training, can we identify the part responsible for each output class (module) in the model(s) and reuse or replace only the desired output classes to build a model? Prior work has proposed decomposing dense-based networks into modules (one for each output class) to enable reusability and replaceability in various scenarios. However, this work is limited to the dense layers and based on the one-to-one relationship between the nodes in consecutive layers. Due to the shared architecture in the CNN model, prior work cannot be adapted directly. In this paper, we propose to decompose a CNN model used for image classification problems into modules for each output class. These modules can further be reused or replaced to build a new model. We have evaluated our approach with CIFAR-10, CIFAR-100, and ImageNet tiny datasets with three variations of ResNet models and found that enabling decomposition comes with a small cost (1.77% and 0.85% for top-1 and top-5 accuracy, respectively). Also, building a model by reusing or replacing modules can be done with a 2.3% and 0.5% average loss of accuracy. Furthermore, reusing and replacing these modules reduces CO2e emission by ~37 times compared to training the model from scratch.

Motivation & Objective

• Address the high computational and environmental cost of retraining CNNs from scratch for new tasks or faulty components.
• Enable modularity in CNNs—previously limited to fully connected networks—by identifying class-specific components.
• Support reuse and replacement of individual modules to fix errors or adapt models without full retraining.
• Reduce carbon emissions associated with deep learning by minimizing redundant training through modular composition.
• Provide a framework for sustainable, incremental model evolution in computer vision applications.

Proposed method

• Train a CNN for image classification and decompose it into modules, where each module is a binary classifier responsible for one output class.
• Use a data-driven approach to identify the activation patterns and parameter contributions specific to each output class across the network.
• Construct each module as a subnetwork that takes the same input size and produces a binary decision (class present or not) using the relevant feature maps and weights.
• Ensure module compatibility by aligning input dimensions and feature hierarchy across modules, enabling reuse across models with similar input specifications.
• Replace or reuse modules in new models by integrating them into a new architecture, preserving the original model's structure and inference pipeline.
• Measure CO₂e emissions using power monitoring tools (e.g., Intel Power Gadget) to compare decomposition-based reuse/replacement with full retraining.

Experimental results

Research questions

• RQ1 Can convolutional neural networks be effectively decomposed into reusable and replaceable modules per output class, despite shared convolutional weights?
• RQ2 What is the accuracy cost of decomposing a trained CNN into class-specific modules compared to the original model?
• RQ3 To what extent can module reuse and replacement improve model performance or fix faulty behavior without retraining?
• RQ4 How much CO₂e emission reduction can be achieved by reusing or replacing modules instead of retraining models from scratch?
• RQ5 What are the practical trade-offs and limitations when applying module reuse and replacement across different datasets and model architectures?

Key findings

• Decomposition incurs a minimal accuracy cost: 1.77% drop in top-1 accuracy and 0.85% in top-5 accuracy on CIFAR-10, CIFAR-100, and ImageNet Tiny.
• Reusing or replacing modules results in a 2.3% average top-1 accuracy loss and 0.5% top-5 loss, demonstrating strong transferability.
• CO₂e emissions are reduced by up to 37 times when reusing or replacing modules instead of retraining from scratch.
• The one-time decomposition overhead generates 116, 371, and 3,800 lbs of CO₂e for CIFAR-10, CIFAR-100, and ImageNet-200, respectively, which is negligible compared to retraining.
• Replacement scenarios reduce CO₂e by 42× (best model) and 31.5× (fixed epoch) compared to training from scratch.
• The method enables practical solutions for fixing faulty classifications—e.g., replacing a faulty gorilla classifier with a corrected one—without retraining the entire model.