[Paper Review] ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation

Adam Paszke, Abhishek Chaurasia+2 more|arXiv (Cornell University)|Jun 7, 2016

Advanced Neural Network ApplicationsComputer Science1,258 citations

TL;DR

ENet is a lightweight encoder-decoder network designed for real-time semantic segmentation with far fewer parameters and FLOPs than prior models, achieving competitive or superior accuracy on Cityscapes, CamVid, and SUN datasets while running on embedded hardware.

ABSTRACT

The ability to perform pixel-wise semantic segmentation in real-time is of paramount importance in mobile applications. Recent deep neural networks aimed at this task have the disadvantage of requiring a large number of floating point operations and have long run-times that hinder their usability. In this paper, we propose a novel deep neural network architecture named ENet (efficient neural network), created specifically for tasks requiring low latency operation. ENet is up to 18$\ imes$ faster, requires 75$\ imes$ less FLOPs, has 79$\ imes$ less parameters, and provides similar or better accuracy to existing models. We have tested it on CamVid, Cityscapes and SUN datasets and report on comparisons with existing state-of-the-art methods, and the trade-offs between accuracy and processing time of a network. We present performance measurements of the proposed architecture on embedded systems and suggest possible software improvements that could make ENet even faster.

Motivation & Objective

• Address the need for real-time pixel-wise semantic segmentation on low-power/mobile devices.
• Develop an efficient encoder-decoder network with a small memory footprint and fast inference.
• Explore design choices that preserve spatial information while maintaining speed.
• Benchmark ENet on Cityscapes, CamVid, and SUN, including embedded hardware performance.

Proposed method

• Introduce ENet architecture with bottleneck blocks and an encoder-decoder structure.
• Use early downsampling with parallel pooling to preserve information flow and improve speed.
• Employ dilated and asymmetric convolutions to enlarge receptive field without excessive computation.
• Replace ReLU with PReLU non-linearities to improve information flow, especially in early layers.
• Apply Spatial Dropout and avoid bias terms in projections to reduce memory/compute.
• Adopt kernel fusion considerations and avoid extensive post-processing to enable end-to-end fast inference.

Experimental results

Research questions

• RQ1 Can ENet achieve real-time semantic segmentation on embedded hardware while maintaining competitive accuracy on standard benchmarks?
• RQ2 What architectural choices (downsampling strategy, dilated/asymmetric convolutions, non-linearities) best balance speed and accuracy for ENet?
• RQ3 How does ENet perform on Cityscapes, CamVid, and SUN compared to SegNet and other baselines?
• RQ4 What are the hardware requirements and potential software limitations affecting ENet’s practical deployment?

Key findings

• ENet achieves substantially lower FLOPs (3.83 GFLOPs) and parameters (0.37M) than SegNet (286.03 GFLOPs, 29.46M parameters), with a model size of about 0.7 MB (fp16).
• On TX1 embedded hardware, ENet runs at 21.1 fps (480×320) and 14.6 fps (640×360), far exceeding SegNet’s speed on the same platform.
• On Titan X, ENet maintains real-time performance with competitive accuracy (Cityscapes: class IoU 58.3 vs SegNet 56.1; Cityscapes category IoU 80.4 vs 79.8).
• Cityscapes test results show ENet achieving higher class IoU and competitive category IoU compared with SegNet, while being the fastest model in the Cityscapes benchmark at the time.
• CamVid results indicate ENet outperforms several baselines on multiple classes, with competitive meanIoU.
• SUN RGB-D results show ENet’s global average and class average accuracy are lower than SegNet, but still offer meaningful real-time performance advantages for RGB data.