[Paper Review] Hierarchically Compositional Tasks and Deep Convolutional Networks

Arturo Deza, Qianli Liao+2 more|arXiv (Cornell University)|Jun 24, 2020

Domain Adaptation and Few-Shot Learning89 references4 citations

TL;DR

This paper investigates why deep convolutional networks (DCNs) outperform fully connected networks on certain visual tasks by examining the role of hierarchical compositional structure in functions. Using deterministic scrambling to disrupt local image structure, the study shows DCNs excel at object recognition—tasks with hierarchical locality—while shallow fully connected networks outperform them on non-compositional tasks like global color estimation, demonstrating that architectural inductive bias must match the task's functional structure.

ABSTRACT

The main success stories of deep learning, starting with ImageNet, depend on deep convolutional networks, which on certain tasks perform significantly better than traditional shallow classifiers, such as support vector machines, and also better than deep fully connected networks; but what is so special about deep convolutional networks? Recent results in approximation theory proved an exponential advantage of deep convolutional networks with or without shared weights in approximating functions with hierarchical locality in their compositional structure. More recently, the hierarchical structure was proved to be hard to learn from data, suggesting that it is a powerful prior embedded in the architecture of the network. These mathematical results, however, do not say which real-life tasks correspond to input-output functions with hierarchical locality. To evaluate this, we consider a set of visual tasks where we disrupt the local organization of images via "deterministic scrambling" to later perform a visual task on these images structurally-altered in the same way for training and testing. For object recognition we find, as expected, that scrambling does not affect the performance of shallow or deep fully connected networks contrary to the out-performance of convolutional networks. Not all tasks involving images are however affected. Texture perception and global color estimation are much less sensitive to deterministic scrambling showing that the underlying functions corresponding to these tasks are not hierarchically local; and also counter-intuitively showing that these tasks are better approximated by networks that are not deep (texture) nor convolutional (color). Altogether, these results shed light into the importance of matching a network architecture with its embedded prior of the task to be learned.

Motivation & Objective

• To identify which real-world visual tasks correspond to functions with hierarchical locality, the key structural property enabling deep convolutional networks' exponential approximation advantage.
• To evaluate whether the success of deep convolutional networks in object recognition stems from their architectural inductive bias toward hierarchical compositionality.
• To test the hypothesis that tasks lacking hierarchical structure—such as global color estimation or texture perception—may be better approximated by non-convolutional or shallower architectures.
• To investigate the limitations of deep convolutional networks by disrupting image structure via deterministic scrambling and measuring performance shifts across architectures.
• To provide empirical validation of theoretical results on approximation efficiency, showing that DCNs are not universally superior but depend on task-specific functional structure.

Proposed method

• Applied deterministic scrambling to images to disrupt local spatial organization while preserving structural consistency for training and testing.
• Trained and evaluated three network architectures: deep convolutional networks (e.g., ThreeConvNet, VGG11), shallow fully connected networks (SFCN), wide fully connected networks (Wide-Net), and deep fully connected networks (Deep-Net) on the same tasks.
• Used mean squared error (MSE) loss to train networks on global color estimation, measuring approximation error across different scrambling levels.
• Compared performance across architectures under varying degrees of image scrambling to assess sensitivity to loss of local structure.
• Leveraged theoretical results from approximation theory to frame the experimental design, particularly the exponential parameter efficiency of deep convolutional networks for hierarchically local functions.
• Used CIFAR-100 dataset for object recognition and color estimation tasks, with unmodified images and scrambled versions for both training and testing.

Experimental results

Research questions

• RQ1 Which visual tasks are inherently hierarchically compositional, and thus benefit from the inductive bias of deep convolutional networks?
• RQ2 How does deterministic scrambling of image structure affect the performance of deep convolutional networks versus fully connected networks on different visual tasks?
• RQ3 Do tasks like global color estimation or texture perception, which lack hierarchical locality, perform better with non-convolutional or shallower architectures?
• RQ4 To what extent does the architectural inductive bias of a network align with the functional structure of the target task?
• RQ5 Can the theoretical advantage of deep convolutional networks in approximating hierarchically local functions be empirically validated through controlled image manipulations?

Key findings

• Deep convolutional networks significantly outperform shallow fully connected networks on object recognition tasks, even after deterministic scrambling, confirming their reliance on hierarchical locality.
• Shallow fully connected networks achieve lower mean squared error in global color estimation than deep convolutional networks, indicating that color estimation is not hierarchically compositional and benefits from simpler architectures.
• Wide fully connected networks outperform both deep fully connected and deep convolutional networks in color estimation, suggesting that increased capacity in non-convolutional architectures can better approximate global, non-compositional functions.
• The performance of deep convolutional networks asymptotes to that of shallow fully connected networks as image scrambling destroys local structure, indicating that DCNs still attempt to exploit locality even when it is absent.
• ThreeConvNet, despite its high performance on texture perception, underperforms relative to fully connected networks on color estimation, confirming that convolutional inductive bias is suboptimal for global, non-local tasks.
• The results empirically validate theoretical claims that deep convolutional networks have an exponential approximation advantage only for functions with hierarchical locality, not for global or non-compositional functions.