[论文解读] The Consciousness Prior

Yoshua Bengio|arXiv (Cornell University)|Sep 25, 2017

Topic Modeling参考文献 19被引用 108

一句话总结

这篇论文提出一种受意识全球工作区理论启发的学习先验，其中一个低维的意识状态从广泛的表示中选择一组稀疏的高层变量，以指导预测、推理和语言化的知识共享。

ABSTRACT

A new prior is proposed for learning representations of high-level concepts of the kind we manipulate with language. This prior can be combined with other priors in order to help disentangling abstract factors from each other. It is inspired by cognitive neuroscience theories of consciousness, seen as a bottleneck through which just a few elements, after having been selected by attention from a broader pool, are then broadcast and condition further processing, both in perception and decision-making. The set of recently selected elements one becomes aware of is seen as forming a low-dimensional conscious state. This conscious state is combining the few concepts constituting a conscious thought, i.e., what one is immediately conscious of at a particular moment. We claim that this architectural and information-processing constraint corresponds to assumptions about the joint distribution between high-level concepts. To the extent that these assumptions are generally true (and the form of natural language seems consistent with them), they can form a useful prior for representation learning. A low-dimensional thought or conscious state is analogous to a sentence: it involves only a few variables and yet can make a statement with very high probability of being true. This is consistent with a joint distribution (over high-level concepts) which has the form of a sparse factor graph, i.e., where the dependencies captured by each factor of the factor graph involve only very few variables while creating a strong dip in the overall energy function. The consciousness prior also makes it natural to map conscious states to natural language utterances or to express classical AI knowledge in a form similar to facts and rules, albeit capturing uncertainty as well as efficient search mechanisms implemented by attention mechanisms.

研究动机与目标

• 激励一种受有意识处理和全球工作区理论启发的机器学习先验。
• 定义一个低维度意识状态 c_t 如何从高层表示 h_t 中选择。
• 论证稀疏因子图模型化高层概念的联合分布。
• 提出将意识状态映射到语言化表示的训练目标和机制。

提出的方法

• 定义 x_t 为观测，h_t 为由表示 RNN 产生的高层无意识表示（h_t = F(x_t, h_{t-1})）。
• 引入通过对 h_t 的注意力得到的意识状态 c_t = C(h_t, c_{t-1}, m_{t-1}, z_t)。
• 使用稀疏因子图建模高层概念的联合分布 P(S) = (∏ f_j(S_j)) / Z。
• 建议一个验证器 V(h_t, c_{t-k})，用于评估当前表示与过去的意识内容之间的一致性。
• 描述训练目标，其中注意力选择信息性元素以预测未来或重建输入，可能使用互信息或多样性项。

实验结果

研究问题

• RQ1 稀疏的、由注意力驱动的意识状态 c_t 是否能改善解耦的高层表示？
• RQ2 将意识状态映射到语言化话语是否有助于解释和学习？
• RQ3 验证器网络是否能可靠判断意识内容与未来状态的一致性？
• RQ4 意识注意目标应如何塑形，以鼓励有用且多样的关注元素？
• RQ5 意识处理是否能提升规划、推理和超长范围预测，超越像素级预测？

主要发现

• 提出意识思想对应于通过注意力选择的高层变量的一个小子集。
• 主张稀疏因子图作为高层概念联合分布的结构形式。
• 认为意识状态可以映射到自然语言话语，从而实现可解释的表示。
• 指出训练可将重建、预测和类似 RL 的目标与一致性验证器相结合。
• 强调通过语言化表示在系统性泛化和组合推理方面的潜在好处。