[论文解读] ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems

Patrice Y. Simard, David Maxwell Chickering+8 more|arXiv (Cornell University)|Sep 16, 2014

Data Stream Mining Techniques参考文献 24被引用 15

一句话总结

ICE 提供了一个可扩展的、交互式的机器学习平台，使非专家能够通过实时人机协作，为大规模、类别不平衡的数据集构建高精度模型。通过利用低延迟、内存内分布式的架构，ICE 加速了用户与模型之间的反馈循环，支持通过直观交互对任务和特征进行迭代优化，显著减少了标注工作量，同时提升了模型质量。

ABSTRACT

Quick interaction between a human teacher and a learning machine presents numerous benefits and challenges when working with web-scale data. The human teacher guides the machine towards accomplishing the task of interest. The learning machine leverages big data to find examples that maximize the training value of its interaction with the teacher. When the teacher is restricted to labeling examples selected by the machine, this problem is an instance of active learning. When the teacher can provide additional information to the machine (e.g., suggestions on what examples or predictive features should be used) as the learning task progresses, then the problem becomes one of interactive learning. To accommodate the two-way communication channel needed for efficient interactive learning, the teacher and the machine need an environment that supports an interaction language. The machine can access, process, and summarize more examples than the teacher can see in a lifetime. Based on the machine's output, the teacher can revise the definition of the task or make it more precise. Both the teacher and the machine continuously learn and benefit from the interaction. We have built a platform to (1) produce valuable and deployable models and (2) support research on both the machine learning and user interface challenges of the interactive learning problem. The platform relies on a dedicated, low-latency, distributed, in-memory architecture that allows us to construct web-scale learning machines with quick interaction speed. The purpose of this paper is to describe this architecture and demonstrate how it supports our research efforts. Preliminary results are presented as illustrations of the architecture but are not the primary focus of the paper.

研究动机与目标

• 解决在专家标注成本高、耗时长的情况下，为大规模、类别不平衡的数据集构建高精度机器学习模型的挑战。
• 设计一种交互式学习系统，支持非专家通过直观的实时反馈引导模型开发。
• 构建可扩展的基础设施，实现实时、低延迟的人机交互。
• 支持交互式学习中机器学习与用户界面挑战的科研工作。
• 通过系统动态选择高价值样本并整合用户建议，减轻标注负担。

提出的方法

• 该平台采用专用的、低延迟、分布式、内存内架构，实现用户与学习系统之间的实时交互。
• 通过允许用户在训练过程中标注样本、提出特征或优化任务定义，支持主动学习与交互式学习。
• 系统持续处理并汇总大规模数据集，仅向用户展示最具信息量的样本以供标注或反馈。
• 机器从用户交互中学习，并迭代更新模型，每次反馈循环后性能均得到提升。
• 交互语言设计支持双向通信，使用户能够有效引导模型的学习过程。
• 该平台旨在同时支持模型部署以及人机协作在机器学习中的科研工作。

实验结果

研究问题

• RQ1 非专家如何在最小化标注工作量的前提下，高效参与大规模、类别不平衡数据集的模型开发？
• RQ2 何种架构设计能够实现在网络规模下，用户与机器学习系统之间的实时、低延迟交互？
• RQ3 与传统主动学习相比，通过用户反馈的交互式学习如何提升模型质量？
• RQ4 系统响应速度在模型开发过程中对维持用户参与度起到何种作用？
• RQ5 可扩展的基础设施如何同时支持模型训练与交互式用户反馈？

主要发现

• ICE 平台使非专家能够通过交互式反馈循环，为大规模、类别不平衡的数据集构建可部署的高质量模型。
• 低延迟、内存内架构实现了用户反馈与模型更新之间的快速迭代，显著缩短了模型构建时间。
• 系统通过优先处理高信息量样本减少标注工作量，提升了学习效率。
• 用户能够动态优化任务定义并提出特征建议，从而获得更精确、更准确的模型。
• 初步结果表明，该平台有效支持人机协作，通过迭代交互实现了可测量的模型性能提升。
• 该平台成功支持了交互式学习中机器学习与用户界面挑战的科研工作。