[论文解读] FasterSeg: Searching for Faster Real-time Semantic Segmentation

Wuyang Chen, Xinyu Gong+4 more|arXiv (Cornell University)|Dec 23, 2019

Advanced Neural Network Applications参考文献 42被引用 114

一句话总结

FasterSeg 自动设计多分辨率 NAS 用于实时语义分割，在 Cityscapes 上比最近的手工基线快超过 30%，同时保持有竞争力的精度。它引入了解耦的延迟正则化及教师-学生协同搜索扩展。

ABSTRACT

We present FasterSeg, an automatically designed semantic segmentation network with not only state-of-the-art performance but also faster speed than current methods. Utilizing neural architecture search (NAS), FasterSeg is discovered from a novel and broader search space integrating multi-resolution branches, that has been recently found to be vital in manually designed segmentation models. To better calibrate the balance between the goals of high accuracy and low latency, we propose a decoupled and fine-grained latency regularization, that effectively overcomes our observed phenomenons that the searched networks are prone to "collapsing" to low-latency yet poor-accuracy models. Moreover, we seamlessly extend FasterSeg to a new collaborative search (co-searching) framework, simultaneously searching for a teacher and a student network in the same single run. The teacher-student distillation further boosts the student model's accuracy. Experiments on popular segmentation benchmarks demonstrate the competency of FasterSeg. For example, FasterSeg can run over 30% faster than the closest manually designed competitor on Cityscapes, while maintaining comparable accuracy.

研究动机与目标

• Motivate real-time semantic segmentation with strict latency constraints.
• Develop a NAS framework that leverages multi-resolution branches inspired by hand-crafted efficient models.
• Address architecture collapse via fine-grained latency regularization during NAS search.
• Extend FasterSeg to a teacher-student co-searching framework to improve student accuracy.
• Demonstrate state-of-the-art speed and competitive accuracy on Cityscapes and transferability to CamVid and BDD.

提出的方法

• Propose a novel search space with multiple resolution branches that can be adaptively selected and aggregated by a head module.
• Introduce a zoomed convolution variant (downsample + conv + upsample) to achieve larger receptive fields with lower latency.
• Use a differentiable superkernel to search expansion ratios within a single operator, enabling flexible width optimization.
• Apply a continuous relaxation of the search space with learnable arch params (alphas, betas, gammas) and Gumbel-Softmax to sample architectures.
• Develop a fine-grained latency regularization that decouples latency across operator, downsample rate, and expansion ratio to prevent architecture collapse.
• Extend to a teacher-student co-searching framework where a teacher and a student are optimized in one run and distilled during training to improve student accuracy.
• Derive a discrete architecture by selecting the best operators and expansion ratios, and determining branch downsampling positions from learned probabilities.

实验结果

研究问题

• RQ1 Can a multi-resolution NAS search space improve real-time segmentation by leveraging branch diversity similar to hand-crafted models?
• RQ2 Does fine-grained latency regularization mitigate architecture collapse during latency-constrained NAS searches?
• RQ3 Can teacher-student co-searching within NAS improve the accuracy of a lightweight student network without increasing latency?
• RQ4 How does FasterSeg perform on standard real-time segmentation benchmarks compared to manually designed architectures?
• RQ5 Is the FasterSeg design transferable across datasets (Cityscapes, CamVid, BDD) without data augmentation beyond standard training?

主要发现

• FasterSeg achieves 73.1% mIoU on Cityscapes val with 163.9 FPS at 1024x2048, outperforming several manual architectures in speed while maintaining competitive accuracy.
• A multi-resolution search space with adaptive branch aggregation is effective for real-time segmentation, and expansion-ratio search (χ) yields faster models without sacrificing accuracy.
• Fine-grained latency regularization successfully addresses architecture collapse, aligning the search with both speed and accuracy goals.
• Teacher-student co-searching yields a higher-accuracy student (73.1% mIoU) than pruning-based distillation, demonstrating the benefit of joint NAS search and knowledge transfer.
• On Cityscapes, FasterSeg outperforms several baselines in FPS (163.9) and achieves competitive mIoU (73.1%), with transferability to CamVid (398.1 FPS) and BDD (318.0 FPS).