[Paper Review] Rethinking Convolutional Semantic Segmentation Learning

Mrinal Haloi|arXiv (Cornell University)|Oct 22, 2017

Retinal Imaging and Analysis22 references3 citations

TL;DR

This paper proposes a joint end-to-end learning framework for convolutional semantic segmentation that simultaneously trains classification and segmentation heads without requiring pre-trained models, using a cooperative learning setup with a novel improved residual inception block and partial attention gating (PAG) for faster convergence and improved accuracy. The method achieves state-of-the-art mIOU of 66.1 on the EyePACS dataset even with random weight initialization, outperforming models requiring ImageNet pretraining.

ABSTRACT

Deep convolutional semantic segmentation (DCSS) learning doesn't converge to an optimal local minimum with random parameters initializations; a pre-trained model on the same domain becomes necessary to achieve convergence.In this work, we propose a joint cooperative end-to-end learning method for DCSS. It addresses many drawbacks with existing deep semantic segmentation learning; the proposed approach simultaneously learn both segmentation and classification; taking away the essential need of the pre-trained model for learning convergence. We present an improved inception based architecture with partial attention gating (PAG) over encoder information. The PAG also adds to achieve faster convergence and better accuracy for segmentation task. We will show the effectiveness of this learning on a diabetic retinopathy classification and segmentation dataset.

Motivation & Objective

• To eliminate the dependency on pre-trained models for convergence in deep convolutional semantic segmentation.
• To address label scarcity in medical imaging by enabling training with partial or random labels for classification.
• To unify the segmentation and classification learning processes into a single end-to-end framework.
• To improve convergence speed and segmentation accuracy using a cooperative learning mechanism between classification and segmentation heads.
• To demonstrate the effectiveness of the method on a real-world biomedical dataset (EyePACS) for diabetic retinopathy detection and microaneurysm segmentation.

Proposed method

• Introduces a cooperative learning framework where segmentation and classification agents share parameters but use separate optimizers and are trained in tandem.
• Employs an improved residual inception block with skip connections and batch normalization to enhance feature learning.
• Applies partial attention gating (PAG) to selectively use mid-level encoder features in the decoder, improving feature representation for segmentation.
• Uses a higher batch size and learning rate for the classifier agent to accelerate information transfer to the segmentation agent.
• Trains the model using Nesterov momentum with a polynomial learning rate decay and batch normalization to reduce covariate shift.
• Leverages a custom framework (TEFLA) built on TensorFlow for training and evaluation.

Experimental results

Research questions

• RQ1 Can deep semantic segmentation converge to an optimal local minimum without pre-trained model initialization?
• RQ2 Can joint training of classification and segmentation improve segmentation performance while reducing reliance on labeled segmentation data?
• RQ3 Can a classification head be effectively trained on partial or random labels without degrading segmentation performance?
• RQ4 Does the proposed cooperative learning framework with partial attention gating lead to faster convergence and higher accuracy than standard encoder-decoder models?
• RQ5 Can the joint model outperform human ophthalmologists in binary diabetic retinopathy classification?

Key findings

• The proposed JointSeg model achieves a mean Intersection over Union (mIOU) of 66.1 on the EyePACS test set, significantly outperforming FCN-8s (55.3) and UNet (64.4) when using pre-trained weights.
• With random weight initialization, FCN-8s and UNet achieve only 15.3 and 25.4 mIOU respectively, demonstrating poor convergence without pre-training, whereas JointSeg maintains 66.1 mIOU.
• The classifier agent achieves 85.1% top-1 accuracy and 88.1% top-2 accuracy on the EyePACS test set, surpassing VGG-19 (82.5% and 85.3%) and InceptionV3 (84.3% and 87.2%).
• In binary diabetic retinopathy classification, the joint model achieves 95.5% accuracy, slightly exceeding human ophthalmologist performance (95.1%).
• Qualitative results show that JointSeg produces more accurate and finer segmentation masks than the improved UNet baseline, especially for small microaneurysms.
• The method enables effective training on partial or random labels for classification without degrading segmentation performance, addressing data scarcity in medical imaging.