[论文解读] KINN: Incorporating Expert Knowledge in Neural Networks

Muhammad Ali Chattha, Shoaib Ahmed Siddiqui+4 more|arXiv (Cornell University)|Feb 15, 2019

Machine Learning and Data Classification参考文献 8被引用 10

一句话总结

KINN 提出了一种新颖的残差知识融合框架，通过将专家预测与深度神经网络（特别是LSTM）相结合，提升模型性能，尤其在数据稀缺条件下表现更优。通过学习对专家输出的修正项而非直接模仿其输出，KINN降低了对数据的依赖，确保了性能下限，并在专家预测存在噪声或错误时仍保持鲁棒性，优于交通流预测中单独使用的模型。

ABSTRACT

The promise of ANNs to automatically discover and extract useful features/patterns from data without dwelling on domain expertise although seems highly promising but comes at the cost of high reliance on large amount of accurately labeled data, which is often hard to acquire and formulate especially in time-series domains like anomaly detection, natural disaster management, predictive maintenance and healthcare. As these networks completely rely on data and ignore a very important modality i.e. expert, they are unable to harvest any benefit from the expert knowledge, which in many cases is very useful. In this paper, we try to bridge the gap between these data driven and expert knowledge based systems by introducing a novel framework for incorporating expert knowledge into the network (KINN). Integrating expert knowledge into the network has three key advantages: (a) Reduction in the amount of data needed to train the model, (b) provision of a lower bound on the performance of the resulting classifier by obtaining the best of both worlds, and (c) improved convergence of model parameters (model converges in smaller number of epochs). Although experts are extremely good in solving different tasks, there are some trends and patterns, which are usually hidden only in the data. Therefore, KINN employs a novel residual knowledge incorporation scheme, which can automatically determine the quality of the predictions made by the expert and rectify it accordingly by learning the trends/patterns from data. Specifically, the method tries to use information contained in one modality to complement information missed by the other. We evaluated KINN on a real world traffic flow prediction problem. KINN significantly superseded performance of both the expert and as well as the base network (LSTM in this case) when evaluated in isolation, highlighting its superiority for the task.

研究动机与目标

• 解决深度神经网络在医疗健康和异常检测等标签数据稀缺领域中对大量数据的高依赖性问题。
• 通过实现两种模态的建设性融合，弥合数据驱动的深度学习与基于专家知识的系统之间的差距。
• 开发一种即使在专家预测不可靠或存在噪声时也能保持性能的框架。
• 通过结合专家模型与神经网络的优势，提供性能下限。
• 通过知识融合学习减少训练时间和数据需求，同时不牺牲准确性。

提出的方法

• KINN 采用残差学习方案，神经网络学习专家预测的修正项，而非学习完整的输入到输出映射。
• 模型同时接收原始输入和专家预测作为输入，其中专家预测被输入到网络中的独立通道。
• 最终预测计算为专家预测与网络学习到的残差之和：$\hat{x}_t = \phi([x_{t-1},...,x_{t-p}, \hat{x}^p_t]; \mathcal{W}) + \text{residual}$。
• 网络使用均方误差（MSE）损失进行端到端训练，使其能够自动通过学习低权重来抑制不可靠的专家输入。
• 该框架具有灵活性，不假设专家预测质量高，当专家知识较差时可自动退化为数据驱动学习。
• 该架构在真实世界交通流预测任务上进行了评估，采用LSTM作为基础网络，并进行了关于数据稀缺性和专家质量的消融研究。

实验结果

研究问题

• RQ1 专家知识能否被有效整合进深度神经网络，以减少对大规模标注数据集的依赖？
• RQ2 当神经网络学习专家预测的残差修正而非直接模仿其输出时，其性能如何变化？
• RQ3 当专家预测存在噪声、错误或完全无信息时会发生什么？
• RQ4 该集成框架能否基于专家和神经网络预测中的最优结果，提供性能下限？
• RQ5 在数据稀缺条件下，残差知识集成方案是否能提升收敛速度和泛化能力？

主要发现

• KINN 在交通流预测中显著优于单独的LSTM和专家模型，实现了最佳整体性能。
• 即使仅使用10%的训练数据，KINN 仍保持稳定性能，得益于数据中强烈的季节性模式，展现出优异的数据效率。
• 当专家预测存在噪声或为零时，KINN 的性能与单独的LSTM相当，证明其能够忽略无用的专家输入。
• 在专家预测存在延迟的情况下，网络学习到为专家通道分配可忽略的权重，实际上退化为标准训练模式。
• KINN 的MSE从未超过两个独立模型（LSTM或专家）中的最优者，证实其实现了两全其美。
• KINN 实现了更快的收敛速度并提升了鲁棒性，尤其在数据稀缺和专家质量差的情况下，验证了其残差集成策略的有效性。