[Paper Review] Learning the Superpixel in a Non-iterative and Lifelong Manner

Lei Zhu, Qi She+5 more|arXiv (Cornell University)|Mar 19, 2021

Medical Image Segmentation Techniques30 references4 citations

TL;DR

This paper proposes LNS-Net, a non-iterative, lifelong learning-based CNN method for unsupervised superpixel generation that eliminates manual labeling and iterative refinement. By using a lightweight feature embedder, non-iterative seed-based clustering, and gradient rescaling to prevent catastrophic forgetting, LNS-Net achieves state-of-the-art performance with 9x lower FLOPs and 20x smaller model size than supervised methods like SSN, while maintaining high contour adherence across diverse datasets.

ABSTRACT

Superpixel is generated by automatically clustering pixels in an image into hundreds of compact partitions, which is widely used to perceive the object contours for its excellent contour adherence. Although some works use the Convolution Neural Network (CNN) to generate high-quality superpixel, we challenge the design principles of these networks, specifically for their dependence on manual labels and excess computation resources, which limits their flexibility compared with the traditional unsupervised segmentation methods. We target at redefining the CNN-based superpixel segmentation as a lifelong clustering task and propose an unsupervised CNN-based method called LNS-Net. The LNS-Net can learn superpixel in a non-iterative and lifelong manner without any manual labels. Specifically, a lightweight feature embedder is proposed for LNS-Net to efficiently generate the cluster-friendly features. With those features, seed nodes can be automatically assigned to cluster pixels in a non-iterative way. Additionally, our LNS-Net can adapt the sequentially lifelong learning by rescaling the gradient of weight based on both channel and spatial context to avoid overfitting. Experiments show that the proposed LNS-Net achieves significantly better performance on three benchmarks with nearly ten times lower complexity compared with other state-of-the-art methods.

Motivation & Objective

• To address the limitations of existing CNN-based superpixel methods that rely on expensive manual annotations and iterative optimization.
• To redefine superpixel segmentation as a lifelong learning task to enable online, sequential model adaptation without catastrophic forgetting.
• To develop a lightweight, unsupervised CNN framework that generates high-quality superpixels efficiently and without iterative refinement.
• To improve generalization across diverse domains, including medical imaging with weak contours and noisy textures.
• To achieve high performance with minimal computational cost and model size, enabling real-time deployment.

Proposed method

• LNS-Net is composed of three modules: a Feature Embedder Module (FEM) for generating cluster-friendly features with low complexity.
• The Non-iterative Clustering Module (NCM) uses a Seed Estimation Layer (SEL) to directly predict optimal cluster centers (seed nodes) and assigns pixels via a cluster layer based on feature similarity.
• A Gradient Rescaling Module (GRM) with Gradient Adaptive Layer (GAL) and Gradient Bi-direction Layer (GBL) dynamically rescales gradients using channel and spatial context to prevent overfitting during lifelong learning.
• A range-limited cluster loss is introduced to train the network in a completely unsupervised manner without requiring ground-truth labels.
• The model is trained sequentially in a lifelong learning fashion, enabling online adaptation to new data streams without retraining from scratch.
• The method avoids iterative cluster refinement by directly estimating seed positions in a single forward pass, enabling real-time inference.

Figure 1: The illustration of the workflow for the proposed LNS-Net, where the top row is the visualization of the features and the bottom is the distribution of the labels jetting in the feature space. The blue ”x” is the generated seed node.

Experimental results

Research questions

• RQ1 Can superpixel segmentation be effectively redefined as a lifelong learning task to enable sequential, online adaptation without manual supervision?
• RQ2 How can a CNN-based superpixel method achieve non-iterative clustering while maintaining high contour adherence and low computational cost?
• RQ3 To what extent can an unsupervised, lightweight CNN outperform existing supervised and unsupervised methods in terms of accuracy and generalization across diverse image domains?
• RQ4 How effective is gradient rescaling based on channel and spatial context in mitigating catastrophic forgetting during lifelong training of superpixel networks?
• RQ5 Can the proposed method generalize well to medical imaging tasks with weak boundaries and high noise, such as retinal and OCT images?

Key findings

• LNS-Net achieves a 0.962 ASA score on BSDS500, outperforming all unsupervised methods and approaching the 0.970 ASA of the supervised SSN method.
• On the DRIVE retinal dataset, LNS-Net achieves the highest ASA (0.953) and second-highest BR (0.833), outperforming even the graph-based ERS method in regularity and speed.
• LNS-Net is 46 times faster than ERS on the DRIVE dataset and 20 times smaller in model size than SSN, with 9x fewer FLOPs.
• On the DME dataset, LNS-Net maintains strong performance despite domain shift and noise, significantly outperforming SSN, which degrades severely under such conditions.
• The model achieves state-of-the-art performance in terms of accuracy and efficiency, with 9x lower FLOPs and 20x smaller model size than SSN, while being fully unsupervised.
• Visual results show that LNS-Net generates more regular superpixels with better contour adherence, especially in complex and noisy regions like retinal and OCT images.

Figure 2: The training strategy of our LSN-Net and other learning-based superpixel segmentation methods. “Ellipses” with different colors mean different clustering tasks (images). Each “dot” means the parameters of the network during training progress. A. The multi-task learning strategy, which lear