[Paper Review] CMD: a framework for Context-aware Model self-Detoxification

Zecheng Tang, Keyan Zhou+6 more|arXiv (Cornell University)|Aug 16, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes CMD, a context-aware framework for self-detoxification of large language models (LLMs) through a step-by-step process called Detox-Chain. By decomposing detoxification into ordered sub-steps—detecting toxicity, masking toxic spans, and rephrasing into non-toxic prompts—CMD enables LLMs to generate safely without compromising coherence or semantic quality. The method achieves significant improvements in both detoxification and generation across six LLMs (1B–33B).

ABSTRACT

Text detoxification aims to minimize the risk of language models producing toxic content. Existing detoxification methods of directly constraining the model output or further training the model on the non-toxic corpus fail to achieve a decent balance between detoxification effectiveness and generation quality. This issue stems from the neglect of constrain imposed by the context since language models are designed to generate output that closely matches the context while detoxification methods endeavor to ensure the safety of the output even if it semantically deviates from the context. In view of this, we introduce a Context-aware Model self-Detoxification~(CMD) framework that pays attention to both the context and the detoxification process, i.e., first detoxifying the context and then making the language model generate along the safe context. Specifically, CMD framework involves two phases: utilizing language models to synthesize data and applying these data for training. We also introduce a toxic contrastive loss that encourages the model generation away from the negative toxic samples. Experiments on various LLMs have verified the effectiveness of our MSD framework, which can yield the best performance compared to baselines.

Motivation & Objective

• Address the conflict between toxic prompt generation and detoxification in autoregressive LLMs.
• Overcome the degradation in generation quality caused by single-step detoxification methods.
• Maintain strong reasoning and generation capabilities while enabling effective toxicity mitigation.
• Develop a structured, multi-step process that guides LLMs to detoxify inputs before generating.
• Enable safe, coherent, and semantically consistent generation even under toxic input conditions.

Proposed method

• Decompose the detoxification process into three ordered sub-steps: toxic content detection, masking of toxic spans, and rephrasing into non-toxic alternatives.
• Introduce Detox-Chain, a chain-of-thought (CoT)-inspired framework that sequences these sub-steps in a logical, step-by-step manner.
• Fine-tune LLMs on a multi-task dataset combining toxic detection, span repair, and continual generation to enhance detoxification capabilities.
• Utilize S-adapter for parameter-efficient fine-tuning, enabling adaptation across diverse LLM architectures (e.g., GPT-2, LLaMA, Flan-T5).
• Leverage open-source models or commercial APIs to generate training data and chains, ensuring scalability and accessibility.
• Ensure the rephrased prompt maintains sufficient context to support coherent continuation generation.

(a) Semantic similarity proportion

Experimental results

Research questions

• RQ1 Can a step-by-step detoxification process reduce the conflict between toxic prompt generation and safety enforcement in LLMs?
• RQ2 How does decomposing detoxification into sub-steps affect generation quality compared to single-step methods?
• RQ3 To what extent can LLMs be trained to detect, mask, and rephrase toxic content in a coherent and context-aware manner?
• RQ4 Does the Detox-Chain framework preserve semantic consistency and discourse coherence during generation after input detoxification?
• RQ5 Can the proposed method be generalized across diverse LLM architectures and model sizes (1B to 33B) without performance degradation?

Key findings

• CMD significantly reduces toxicity in model generations while preserving semantic consistency and discourse coherence across six LLMs (1B–33B).
• Automatic and human evaluations show that models trained with Detox-Chain outperform baseline single-step detoxification methods in both toxicity reduction and generation quality.
• The Detox-Chain framework enables models to maintain strong reasoning and generation capabilities even when processing initially toxic inputs.
• Models fine-tuned with the multi-task training setup (toxic detection, span repair, generation) show improved performance on both safety and fluency metrics.
• The method achieves measurable improvements in semantic similarity (SIM) and reduced perplexity (PPL), indicating better alignment between input and output.
• The framework is effective across diverse architectures, including GPT-2-XL, LLaMA, Alpaca, and Flan-T5, demonstrating broad applicability.

(b) Toxicity of LLMs (SIM)