[Paper Review] Complexity of Representation and Inference in Compositional Models with Part Sharing

Alan Yuille, Roozbeh Mottaghi|arXiv (Cornell University)|Jan 16, 2013

Neural Networks and Applications30 references3 citations

TL;DR

This paper analyzes the computational complexity of compositional models with part sharing, showing that part sharing enables linear-time inference in serial computers when dictionary size grows exponentially with hierarchy level, and linear-time parallel inference with exponential node counts when dictionary size decreases exponentially. The model uses hierarchical dictionaries and executive-summary principles to reduce representation and inference costs.

ABSTRACT

This paper describes serial and parallel compositional models of multiple objects with part sharing. Objects are built by part-subpart compositions and expressed in terms of a hierarchical dictionary of object parts. These parts are represented on lattices of decreasing sizes which yield an executive summary description. We describe inference and learning algorithms for these models. We analyze the complexity of this model in terms of computation time (for serial computers) and numbers of nodes (e.g., "neurons") for parallel computers. In particular, we compute the complexity gains by part sharing and its dependence on how the dictionary scales with the level of the hierarchy. We explore three regimes of scaling behavior where the dictionary size (i) increases exponentially with the level, (ii) is determined by an unsupervised compositional learning algorithm applied to real data, (iii) decreases exponentially with scale. This analysis shows that in some regimes the use of shared parts enables algorithms which can perform inference in time linear in the number of levels for an exponential number of objects. In other regimes part sharing has little advantage for serial computers but can give linear processing on parallel computers.

Motivation & Objective

• To analyze the computational complexity of compositional models with part sharing in both serial and parallel computing settings.
• To investigate how dictionary scaling across hierarchical levels affects inference efficiency and representation cost.
• To evaluate the benefits of part sharing and executive-summary principles in reducing computational burden for object recognition.
• To explore the implications of these complexity results for biological and artificial vision systems, particularly in relation to cortical architecture.

Proposed method

• Formulates compositional models using hierarchical graphical probability models with recursive part-subpart composition across H levels.
• Represents parts via dictionaries M_h at each level h, where higher-level objects are built from lower-level parts using shared components.
• Applies the executive-summary principle to coarsely represent part states based on subpart states, reducing computational load.
• Re-formulates models on regular lattices to enable comparison with deep belief networks and to support parallel implementation.
• Analyzes complexity in terms of computation time (serial) and number of nodes (parallel), using mathematical analysis of dictionary size scaling.
• Considers three scaling regimes: exponential increase, unsupervised learning-based, and exponential decrease in dictionary size with level.

Experimental results

Research questions

• RQ1 How does part sharing affect inference time in serial versus parallel computing architectures?
• RQ2 What is the impact of dictionary size scaling across hierarchical levels on computational complexity?
• RQ3 In what regimes does part sharing yield significant complexity gains for inference?
• RQ4 How does the executive-summary principle reduce representation and inference costs in compositional models?
• RQ5 What implications do these complexity results have for modeling visual cortex architecture, particularly in V1/V2 versus IT areas?

Key findings

• When dictionary size grows exponentially with hierarchy level (|M_h| = a/q^h), part sharing enables inference in time linear in the number of levels H, even for an exponential number of objects.
• In the regime where dictionary size is determined by unsupervised learning on real data, part sharing provides significant complexity gains for serial inference.
• When dictionary size decreases exponentially with level (|M_h| = r^{H-h}), part sharing offers no advantage for serial computers but enables linear-time inference on parallel computers using exponentially many nodes.
• The executive-summary principle allows coarse representation of part states based on subpart states, reducing the need for high-resolution detail during inference.
• The results suggest that large visual areas like V1 and V2 may serve as high-capacity dictionaries for low-level features, explaining their disproportionate size in the cortex.
• Parallel inference algorithms resemble bottom-up processing followed by top-down context resolution, aligning with neurobiological models of visual perception.