[Paper Review] The Heterogeneity Hypothesis: Finding Layer-Wise Dissimilated Network Architecture.

Yawei Li, Wen Li+5 more|arXiv (Cornell University)|Jun 29, 2020

Advanced Neural Network Applications26 references4 citations

TL;DR

This paper introduces the 'heterogeneity hypothesis,' proposing that layer-wise dissimilated network architectures (LW-DNA) — created by pruning widened baseline networks — achieve superior performance over standard architectures with lower model complexity. The method identifies optimal, layer-specific channel configurations without additional training cost, consistently outperforming baselines across image classification, visual tracking, and image restoration tasks.

ABSTRACT

In this paper, we tackle the problem of convolutional neural network design. Instead of focusing on the overall architecture design, we investigate a design space that is usually overlooked, \ie adjusting the channel configurations of predefined networks. We find that this adjustment can be achieved by pruning widened baseline networks and leads to superior performance. Base on that, we articulate the ``heterogeneity hypothesis'': with the same training protocol, there exists a layer-wise dissimilated network architecture (LW-DNA) that can outperform the original network with regular channel configurations under lower level of model complexity. The LW-DNA models are identified without added computational cost and training time compared with the original network. This constraint leads to controlled experiment which directs the focus to the importance of layer-wise specific channel configurations. Multiple sources of hints relate the benefits of LW-DNA models to overfitting, \ie the relative relationship between model complexity and dataset size. Experiments are conducted on various networks and datasets for image classification, visual tracking and image restoration. The resultant LW-DNA models consistently outperform the compared baseline models.

Motivation & Objective

• To investigate the under-explored design space of channel configuration adjustments in convolutional neural networks.
• To address the limitation of fixed, uniform channel configurations across layers in standard network designs.
• To determine whether layer-wise specific channel configurations can yield better performance with reduced complexity.
• To validate that performance gains stem from improved generalization, particularly in relation to dataset size and model complexity.
• To develop a method for identifying high-performing LW-DNA models without increasing training cost or computational overhead.

Proposed method

• Pruning widened baseline networks to identify layer-wise dissimilated network architectures (LW-DNA) with optimized channel configurations.
• Applying a consistent training protocol across all models to isolate the effect of channel configuration on performance.
• Using the same training setup as the original network to ensure no added training cost or time for LW-DNA identification.
• Analyzing the relationship between model complexity and dataset size to explain performance gains via reduced overfitting.
• Systematically evaluating LW-DNA models across multiple architectures and datasets to validate generalization.
• Leveraging insights from overfitting dynamics to guide the selection of layer-specific channel configurations.

Experimental results

Research questions

• RQ1 Can layer-wise dissimilated network architectures (LW-DNA) achieve better performance than standard architectures with lower model complexity?
• RQ2 Does the performance gain of LW-DNA stem from improved generalization, particularly in relation to dataset size and model complexity?
• RQ3 Can LW-DNA be identified without additional training cost or computational overhead compared to the original network?
• RQ4 How does the relative relationship between model complexity and dataset size influence the effectiveness of LW-DNA?
• RQ5 To what extent do LW-DNA models generalize across different tasks such as image classification, visual tracking, and image restoration?

Key findings

• LW-DNA models consistently outperform standard baseline networks across multiple image classification benchmarks.
• The proposed method identifies high-performing LW-DNA architectures without increasing training time or computational cost.
• Performance gains are attributed to reduced overfitting, especially when model complexity is well-matched to dataset size.
• The heterogeneity hypothesis is validated: a layer-wise dissimilated architecture exists that outperforms the original with lower complexity.
• Experiments across diverse tasks — including visual tracking and image restoration — confirm the robustness and generalization of LW-DNA models.
• The method reveals that optimal channel configurations are layer-specific and cannot be captured by uniform, symmetric designs.