[Paper Review] Multi-Residual Networks.

Masoud Abdi, Saeid Nahavandi|arXiv (Cornell University)|Sep 19, 2016

Adversarial Robustness in Machine Learning13 references16 citations

TL;DR

This paper proposes multi-residual networks, which enhance deep residual networks by increasing the number of residual functions within residual blocks. By doing so, the model achieves improved generalization and classification accuracy, reducing test error to 3.92% on CIFAR-10—outperforming all prior models and demonstrating that deep residual networks function as exponential ensembles by design.

ABSTRACT

In this article, we take one step toward understanding the learning behavior of deep residual networks, and supporting the hypothesis that deep residual networks are exponential ensembles by construction. We examine the effective range of ensembles by introducing multi-residual networks that significantly improve classification accuracy of residual networks. The multi-residual networks increase the number of residual functions in the residual blocks. This is shown to improve the accuracy of the residual network when the network is deeper than a threshold. Based on a series of empirical studies on CIFAR-10 and CIFAR-100 datasets, the proposed multi-residual network yield $6\%$ and $10\%$ improvement with respect to the residual networks with identity mappings. Comparing with other state-of-the-art models, the proposed multi-residual network obtains a test error rate of $3.92\%$ on CIFAR-10 that outperforms all existing models.

Motivation & Objective

• To investigate the learning behavior of deep residual networks and validate the hypothesis that they function as exponential ensembles.
• To address the accuracy degradation in very deep residual networks beyond a certain depth threshold.
• To improve classification performance on standard benchmarks like CIFAR-10 and CIFAR-100 through architectural innovation.
• To empirically demonstrate that increasing residual functions per block enhances model expressiveness and generalization.

Proposed method

• Introduce multi-residual networks by modifying residual blocks to include multiple residual functions instead of a single one.
• Design residual blocks with multiple skip connections and parallel residual paths to increase the effective ensemble size.
• Train the resulting architecture on CIFAR-10 and CIFAR-100 datasets using standard optimization protocols.
• Compare performance against standard residual networks with identity mappings and state-of-the-art models.
• Use empirical analysis to evaluate the effective range of ensemble behavior in deeper networks.
• Analyze the impact of residual function count on generalization and convergence in deep architectures.

Experimental results

Research questions

• RQ1 To what extent do deep residual networks behave as exponential ensembles, and can this be enhanced through architectural changes?
• RQ2 How does increasing the number of residual functions per block affect classification accuracy in very deep networks?
• RQ3 At what depth threshold does the performance gain from multi-residual blocks become significant?
• RQ4 Can multi-residual networks achieve state-of-the-art performance on CIFAR-10 and CIFAR-100 compared to existing models?
• RQ5 What is the empirical evidence supporting the hypothesis that residual networks are exponential ensembles by construction?

Key findings

• The proposed multi-residual network achieves a test error rate of 3.92% on CIFAR-10, outperforming all existing models.
• On CIFAR-100, the model achieves a 10% improvement in accuracy compared to standard residual networks with identity mappings.
• The accuracy gain is most pronounced when the network depth exceeds a certain threshold, indicating a depth-dependent benefit.
• The improvement is attributed to the increased number of residual functions, which enhances the effective ensemble capacity.
• Empirical results support the hypothesis that residual networks are exponential ensembles by construction, with multi-residual networks amplifying this behavior.
• The method significantly improves generalization, especially in deeper architectures, without increasing model complexity unduly.