[论文解读] Adapting Large Language Models for Content Moderation: Pitfalls in Data Engineering and Supervised Fine-tuning

Huan Ma, Changqing Zhang+3 more|arXiv (Cornell University)|Oct 5, 2023

Hate Speech and Cyberbullying Detection被引用 6

一句话总结

本文提出一种针对私有部署大型语言模型（LLMs）在内容审核中的监督微调方法，通过整合如GPT-4等强大LLMs的思维链（reasoning chains），提升模型的鲁棒性与可解释性。该方法显著增强了模型的泛化能力，在实现接近GPT-3.5性能的同时，支持领域特定、合规且成本可控的审核系统。

ABSTRACT

Nowadays, billions of people engage in communication and express their opinions on the internet daily. Unfortunately, not all of these expressions are friendly or compliant, making content moderation an indispensable task. A common approach is to use a discriminative model to classify the content, but this method often requires strict data engineering, otherwise it will face unacceptable overfitting. With the successful development of Large Language Models (LLMs) in recent years, LLM-based methods have become a feasible solution for handling tasks in various domains. Thanks to the knowledge of the foundation models, we can develop more robust privately deployed models with limited data via fine-tuning these foundation models. Moreover, as a generative model, it can provide detailed analysis of the review process, enhancing interpretability. In this paper, we introduce how to fine-tune a LLM model that can be privately deployed for content moderation. Specifically, we discuss the differences between discriminative and generative models using content moderation as an example. Additionally, we reveal that incorporating reasoning processes during the fine-tuning of LLMs can effectively alleviate overfitting, even if the model is not allowed to directly output reasoning processes during deployment. We present a complete process, from data collection and construction to model training and overfitting elimination, for fine-tuning LLMs in vertical domain deployments. We report the entire research process and the key findings in this paper, hoping to provide valuable experience for researchers who are fine-tuning privately deployed models in their domain-specific research.

研究动机与目标

• 开发一种基于微调LLMs的私有化、领域特定内容审核系统，满足合规性与成本约束。
• 探究在微调过程中引入思维链是否相比直接分类能提升模型的鲁棒性与可解释性。
• 评估使用LLM生成的思维链作为弱监督信号对微调性能的影响，尤其在人工标注链稀缺时的表现。
• 分析SFT中的数据工程选择，包括如何处理源LLM输出的错误或不一致思维链。
• 提供一个系统化、可复现的框架，用于在真实世界内容审核场景中部署微调后的LLMs。

提出的方法

• 使用包含审核标签与由GPT-4等强大LLM生成的思维链（思维链）的训练数据，对私有部署的LLM进行监督微调（SFT）。
• 通过利用GPT-4为训练数据生成思维过程，实现弱监督，降低对昂贵人工标注思维链的依赖。
• 设计数据工程流水线，以处理GPT-4输出的错误或不一致思维链，评估多种过滤与修正策略。
• 对比带思维链与不带思维链的SFT，评估其对模型泛化能力与分布外鲁棒性的影响。
• 实施两阶段流程：首先，使用GPT-4生成带思维链的训练数据；其次，使用标准SFT目标在该数据上微调目标LLM。
• 在标准内容审核基准上评估模型性能，测量准确率、鲁棒性以及输出的可解释性。

Figure 1: Content Moderation with Auditing Processes. GPT-4 can provide complete auditing processes (left), but sometimes it presents limitations (right).

实验结果

研究问题

• RQ1 在微调过程中引入思维链是否能提升私有部署LLMs在内容审核中的泛化能力与鲁棒性？
• RQ2 当人工标注的思维链稀缺时，使用GPT-4生成的思维链作为弱监督信号，对微调性能有何影响？
• RQ3 源LLM输出的错误或幻觉性思维链对最终微调模型性能有何影响？
• RQ4 处理有缺陷思维链的不同数据工程策略如何影响模型的可靠性与可解释性？
• RQ5 当使用带思维链的数据进行训练时，私有部署的LLM是否能在内容审核任务中达到与GPT-3.5相当的性能？

主要发现

• 与直接分类的微调相比，使用思维链进行微调显著提升了模型的泛化能力与分布外鲁棒性。
• 利用GPT-4生成的思维链作为弱监督信号，可在极低人工标注成本下实现高性能，大幅降低数据准备成本。
• GPT-4输出的错误思维链会降低模型性能，但通过精心设计的数据过滤与修正策略可有效缓解其负面影响。
• 经过思维链增强微调的模型在内容审核基准上的表现与GPT-3.5相当，展现出强大的竞争力。
• 引入思维过程显著提升了可解释性，使审核决策过程可透明审计。
• 本研究证明，思维链增强的SFT是一种可行、可扩展且合规的替代方案，可避免依赖第三方LLMs进行内容审核。

Figure 2: Framework of fine-tuing privately deployed content moderation model. Fine-tuning LLMs with the pattern $(x_{i},y_{i})$ can lead to severe overfitting. For instance, sentences that contain regional characteristics will be classified into Discrimination or Insult. Incorporating an auditing p