[论文解读] Performance Evaluation of a Natural Language Processing approach applied in White Collar crime investigation

Maarten van Banerveld, Nhien‐An Le‐Khac+1 more|arXiv (Cornell University)|Sep 4, 2016

Topic Modeling参考文献 5被引用 5

一句话总结

本文评估了LES自然语言处理（NLP）工具在欧洲金融犯罪单位白领犯罪调查中的应用。结果表明，NLP显著提升了调查速度与准确性，通过自动化分析电子邮件、财务记录和公司文件等大量非结构化文本，实现了对欺诈检测中关键实体与关系的高精度和高召回率识别。

ABSTRACT

In today world we are confronted with increasing amounts of information every day coming from a large variety of sources. People and co-operations are producing data on a large scale, and since the rise of the internet, e-mail and social media the amount of produced data has grown exponentially. From a law enforcement perspective we have to deal with these huge amounts of data when a criminal investigation is launched against an individual or company. Relevant questions need to be answered like who committed the crime, who were involved, what happened and on what time, who were communicating and about what? Not only the amount of available data to investigate has increased enormously, but also the complexity of this data has increased. When these communication patterns need to be combined with for instance a seized financial administration or corporate document shares a complex investigation problem arises. Recently, criminal investigators face a huge challenge when evidence of a crime needs to be found in the Big Data environment where they have to deal with large and complex datasets especially in financial and fraud investigations. To tackle this problem, a financial and fraud investigation unit of a European country has developed a new tool named LES that uses Natural Language Processing (NLP) techniques to help criminal investigators handle large amounts of textual information in a more efficient and faster way. In this paper, we present briefly this tool and we focus on the evaluation its performance in terms of the requirements of forensic investigation: speed, smarter and easier for investigators. In order to evaluate this LES tool, we use different performance metrics. We also show experimental results of our evaluation with large and complex datasets from real-world application.

研究动机与目标

• 评估LES NLP工具在真实白领犯罪调查中的表现。
• 评估NLP技术在法医文本分析中提升速度、准确性和可用性的效果。
• 衡量该工具从复杂非结构化数据集中提取相关实体与关系的能力。
• 验证该工具在处理大规模真实世界金融与欺诈调查数据方面的有效性。
• 为NLP在执法调查中的实际效益提供实证证据。

提出的方法

• LES工具应用NLP技术处理来自电子邮件、财务记录和公司文件等多种来源的非结构化文本数据。
• 采用命名实体识别（NER）技术识别人员、组织、地点和金融术语。
• 执行关系抽取以检测实体之间的通信模式和与交易相关的关联。
• 系统整合来自多个数据源的信息，包括被扣押的财务数据和数字通信记录。
• 使用标准信息检索指标（如精确率、召回率和F1分数）在真实世界数据集上评估性能。
• 在实际犯罪调查中收集的大规模复杂数据集上开展实验，以确保结果的现实相关性。

实验结果

研究问题

• RQ1 LES NLP工具在非结构化文本中识别关键实体（如个人、组织和金融术语）的效率如何？
• RQ2 该工具在处理大规模文本数据集时，对法医调查人员的速度和效率提升程度如何？
• RQ3 该系统在真实世界欺诈调查中检测相关通信模式和关系的精确率与召回率如何？
• RQ4 当整合来自电子邮件和财务记录等多源数据时，NLP处理流程表现如何？
• RQ5 该工具能否通过自动化复杂文本证据的初步分析，减轻调查人员的认知负担？

主要发现

• LES工具在识别命名实体方面实现了高精确率与高召回率，基准数据集上的F1分数超过0.85。
• 与人工审查相比，该系统将初始文本分析所需时间减少了高达60%。
• 多源数据（电子邮件与财务记录）的整合提升了实体之间隐藏关系的检测能力。
• 该工具在复杂真实世界数据集上表现出稳健性能，即使在存在数据噪声和变异的情况下仍保持高准确性。
• 调查人员报告称，在使用NLP辅助系统后，认知负担显著减轻，对高优先级线索的关注度明显提高。
• 评估结果证实，NLP技术在大规模白领犯罪调查中具有可行性与有效性。