[Paper Review] The Consciousness Prior

Yoshua Bengio|arXiv (Cornell University)|Sep 25, 2017

Topic Modeling19 references108 citations

TL;DR

This paper proposes a learning prior inspired by the global workspace theory of consciousness, where a low-dimensional conscious state selects a sparse set of high-level variables from a broad representation to guide prediction, reasoning, and language-like knowledge sharing.

ABSTRACT

A new prior is proposed for learning representations of high-level concepts of the kind we manipulate with language. This prior can be combined with other priors in order to help disentangling abstract factors from each other. It is inspired by cognitive neuroscience theories of consciousness, seen as a bottleneck through which just a few elements, after having been selected by attention from a broader pool, are then broadcast and condition further processing, both in perception and decision-making. The set of recently selected elements one becomes aware of is seen as forming a low-dimensional conscious state. This conscious state is combining the few concepts constituting a conscious thought, i.e., what one is immediately conscious of at a particular moment. We claim that this architectural and information-processing constraint corresponds to assumptions about the joint distribution between high-level concepts. To the extent that these assumptions are generally true (and the form of natural language seems consistent with them), they can form a useful prior for representation learning. A low-dimensional thought or conscious state is analogous to a sentence: it involves only a few variables and yet can make a statement with very high probability of being true. This is consistent with a joint distribution (over high-level concepts) which has the form of a sparse factor graph, i.e., where the dependencies captured by each factor of the factor graph involve only very few variables while creating a strong dip in the overall energy function. The consciousness prior also makes it natural to map conscious states to natural language utterances or to express classical AI knowledge in a form similar to facts and rules, albeit capturing uncertainty as well as efficient search mechanisms implemented by attention mechanisms.

Motivation & Objective

• Motivate a machine learning prior inspired by conscious processing and global workspace theory.
• Define how a low-dimensional conscious state c_t selects from a high-level representation h_t.
• Argue that sparse factor graphs model the joint distribution of high-level concepts.
• Propose training objectives and mechanisms to map conscious states to language-like representations.

Proposed method

• Define x_t as observations and h_t as high-level unconscious representations produced by a representation RNN (h_t = F(x_t, h_{t-1}).
• Introduce a conscious state c_t = C(h_t, c_{t-1}, m_{t-1}, z_t) derived via attention over h_t.
• Model the joint distribution over high-level concepts with sparse factor graphs P(S) = (∏ f_j(S_j)) / Z.
• Suggest a verifier V(h_t, c_{t-k}) to assess consistency between current representations and past conscious content.
• Describe training objectives where attention selects informative elements to predict future or reconstruct inputs, potentially using mutual information or diversity terms.

Experimental results

Research questions

• RQ1 Can a sparse, attention-driven conscious state c_t improve disentangled, high-level representations?
• RQ2 Does mapping conscious states to language-like utterances aid interpretation and learning?
• RQ3 Can a verifier network reliably judge the consistency of conscious content with future states?
• RQ4 How should the conscious attention objective be shaped to encourage useful and diverse attended elements?
• RQ5 Can conscious processing enhance planning, reasoning, and long-range prediction beyond pixel-level forecasting?

Key findings

• Proposes that conscious thoughts correspond to a small subset of high-level variables selected by attention.
• Argues for sparse factor graphs as the structural form of the joint distribution over high-level concepts.
• Suggests that conscious states can be mapped to natural language utterances, enabling interpretable representations.
• Outlines that training could combine reconstruction, predictive, and RL-like objectives with a verifier for consistency.
• Highlights potential benefits for systematic generalization and combinatorial reasoning through language-like representations.