[Paper Review] Light-Weight RefineNet for Real-Time Semantic Segmentation

Vladimir Nekrasov, Chunhua Shen+1 more|arXiv (Cornell University)|Oct 8, 2018

Advanced Neural Network ApplicationsComputer Science57 references101 citations

TL;DR

The paper adapts RefineNet into a compact, real-time semantic segmentation model by removing redundant blocks and replacing 3x3 convolutions with 1x1 convolutions, achieving over 2x parameter reduction with minimal accuracy loss and up to 55 FPS on 512×512 inputs.

ABSTRACT

We consider an important task of effective and efficient semantic image segmentation. In particular, we adapt a powerful semantic segmentation architecture, called RefineNet, into the more compact one, suitable even for tasks requiring real-time performance on high-resolution inputs. To this end, we identify computationally expensive blocks in the original setup, and propose two modifications aimed to decrease the number of parameters and floating point operations. By doing that, we achieve more than twofold model reduction, while keeping the performance levels almost intact. Our fastest model undergoes a significant speed-up boost from 20 FPS to 55 FPS on a generic GPU card on 512x512 inputs with solid 81.1% mean iou performance on the test set of PASCAL VOC, while our slowest model with 32 FPS (from original 17 FPS) shows 82.7% mean iou on the same dataset. Alternatively, we showcase that our approach is easily mixable with light-weight classification networks: we attain 79.2% mean iou on PASCAL VOC using a model that contains only 3.3M parameters and performs only 9.3B floating point operations.

Motivation & Objective

• Motivate real-time semantic segmentation on high-resolution inputs.
• Identify and remove computational bottlenecks in RefineNet without substantial accuracy loss.
• Demonstrate that encoder-decoder style segmentation can be accelerated with lightweight decoder blocks.
• Show compatibility with lightweight backbones (NASNet-Mobile, MobileNet-v2) and other backbones.

Proposed method

• Replace expensive 3x3 convolutions in CRP and fusion blocks with 1x1 convolutions to reduce parameters and FLOPs.
• Adopt bottleneck design for residual convolutions (RCU) and remove RCU blocks when redundant.
• Retain CRP blocks with 5x5 pooling to preserve context and empirical receptive field.
• Demonstrate that omitting RCU blocks does not degrade performance in the light-weight regime.
• Validate architecture across multiple backbones (ResNet-50/101/152, NASNet-Mobile, MobileNet-v2).

Experimental results

Research questions

• RQ1 Can RefineNet be restructured to meet real-time constraints without major accuracy loss?
• RQ2 Which components of RefineNet are essential for performance, and which can be removed or simplified?
• RQ3 How does the light-weight RefineNet perform across different backbones and datasets?
• RQ4 Is replacing 3x3 convolutions with 1x1 convolutions sufficient to preserve contextual information in the decoder?
• RQ5 Can the method leverage lightweight backbones while maintaining competitive segmentation quality?

Key findings

• Twofold reduction in model size and substantial FLOP reduction while maintaining competitive mean IoU.
• Fastest LW-RefineNet achieves 55 FPS on 512×512 inputs with 81.1% mIoU on PASCAL VOC test set.
• Slowest LW-RefineNet (still faster than original) achieves 82.7% mIoU on VOC with 32 FPS.
• Light-weight variants using NASNet-Mobile and MobileNet-v2 achieve competitive VOC results with drastically fewer parameters and FLOPs.
• Ablations show CRP dominates segmentation performance, while removing RCU blocks yields minimal or no loss in the light-weight regime.
• The approach is compatible with various backbones and can be complemented by additional model compression techniques.