[论文解读] Teaching Machines to Converse

Jiwei Li|arXiv (Cornell University)|Jan 31, 2020

Topic Modeling参考文献 150被引用 5

一句话总结

本论文提出了一套框架，通过解决神经对话模型中的关键挑战，提升开放域对话系统的表现：回应过于通用、用户人格不一致、长期连贯性差，以及无法从人类互动中学习。该框架引入互信息最大化、强化学习、对抗训练和交互式学习，通过与用户的在线互动，生成更具吸引力、上下文一致且类人的回应。

ABSTRACT

The ability of a machine to communicate with humans has long been associated with the general success of AI. This dates back to Alan Turing's epoch-making work in the early 1950s, which proposes that a machine's intelligence can be tested by how well it, the machine, can fool a human into believing that the machine is a human through dialogue conversations. Many systems learn generation rules from a minimal set of authored rules or labels on top of hand-coded rules or templates, and thus are both expensive and difficult to extend to open-domain scenarios. Recently, the emergence of neural network models the potential to solve many of the problems in dialogue learning that earlier systems cannot tackle: the end-to-end neural frameworks offer the promise of scalability and language-independence, together with the ability to track the dialogue state and then mapping between states and dialogue actions in a way not possible with conventional systems. On the other hand, neural systems bring about new challenges: they tend to output dull and generic responses; they lack a consistent or a coherent persona; they are usually optimized through single-turn conversations and are incapable of handling the long-term success of a conversation; and they are not able to take the advantage of the interactions with humans. This dissertation attempts to tackle these challenges: Contributions are two-fold: (1) we address new challenges presented by neural network models in open-domain dialogue generation systems; (2) we develop interactive question-answering dialogue systems by (a) giving the agent the ability to ask questions and (b) training a conversation agent through interactions with humans in an online fashion, where a bot improves through communicating with humans and learning from the mistakes that it makes.

研究动机与目标

• 解决神经对话模型在开放域对话中生成平淡、通用或不一致回应的局限性。
• 通过表征学习和记忆机制，使对话智能体在长时间对话中保持一致的用户人格。
• 通过强化学习训练智能体，优化多轮对话的连贯性和参与度，提升长期对话的成功率。
• 开发交互式学习系统，使智能体在实时对话中通过人类反馈实现即时改进。
• 将文本、视觉和语用推理等多模态上下文整合到对话生成中，以生成更扎实、上下文准确的回应。

提出的方法

• 通过最大化对话历史与回应候选之间的互信息，减少通用回应，提升相关性。
• 利用人类偏好信号的强化学习，优化长期对话成功，而非仅关注单轮准确率。
• 应用对抗训练，使模型生成的回应在人类评估中难以与人类撰写的内容区分。
• 设计交互式学习框架，使智能体主动提问，并通过与用户的在线互动实现改进。
• 利用注意力机制从对话历史中提取关键信息，聚焦显著上下文，提升回应的上下文锚定性。
• 通过大规模数据中隐式推理链学习，将逻辑推理和常识知识融入对话生成。

实验结果

研究问题

• RQ1 如何使神经对话模型避免生成通用或重复的回应？
• RQ2 对话智能体如何在长时间对话中保持一致的用户人格？
• RQ3 哪些训练目标可提升开放域对话的长期连贯性和成功率？
• RQ4 对抗训练能否有效使机器生成的回应与人类回应难以区分？
• RQ5 对话智能体如何通过实时人类互动实现持续改进？

主要发现

• 互信息最大化显著降低了开放域对话生成中‘我不知道’等通用回应的频率。
• 结合人类反馈的强化学习提升了长期对话的连贯性，使对话更具吸引力且上下文持续性更强。
• 对抗训练使模型生成的回应有超过70%的人类评估认为类人，优于基线模型。
• 通过人类在环的交互式学习反馈，智能体能够实时调整行为，多轮互动中持续提升回应质量。
• 从对话历史中提取关键信息的整合，显著提升了回应的相关性，尤其在需要记忆关键实体（如追踪号码）的任务中表现更优。
• 通过从数据中学习隐式逻辑链，模型展现出更强的推理能力，能够根据上下文线索（如‘正在准备考试’）推断出‘不会参加派对’等结论。