[Paper Review] Reward Modeling for Mitigating Toxicity in Transformer-based Language Models

Farshid Faal, Ketra Schmitt+1 more|arXiv (Cornell University)|Feb 19, 2022

Topic ModelingComputer Science43 references28 citations

TL;DR

This paper proposes Reinforce-Detoxify, a reinforcement learning-based method that uses a multitask learning (MTL)-based reward model to reduce toxicity in transformer-based language models while minimizing unintended bias toward marginalized social identities. By fine-tuning the model via proximal policy optimization (PPO) with a KL divergence penalty, it achieves state-of-the-art performance in toxicity reduction without sacrificing fluency or coverage of identity-related content.

ABSTRACT

Transformer-based language models are able to generate fluent text and be efficiently adapted across various natural language generation tasks. However, language models that are pretrained on large unlabeled web text corpora have been shown to suffer from degenerating toxic content and social bias behaviors, consequently hindering their safe deployment. Various detoxification methods were proposed to mitigate the language model's toxicity; however, these methods struggled to detoxify language models when conditioned on prompts that contain specific social identities related to gender, race, or religion. In this study, we propose Reinforce-Detoxify; A reinforcement learning-based method for mitigating toxicity in language models. We address the challenge of safety in language models and propose a new reward model that is able to detect toxic content and mitigate unintended bias towards social identities in toxicity prediction. The experiments demonstrate that the Reinforce-Detoxify method for language model detoxification outperforms existing detoxification approaches in automatic evaluation metrics, indicating the ability of our approach in language model detoxification and less prone to unintended bias toward social identities in generated content.

Motivation & Objective

• To address the unintended bias in existing detoxification methods that disproportionately suppress language from marginalized social identities such as Black, Asian, or Muslim communities.
• To develop a reward model that detects toxicity while mitigating racial, gender, and religious biases in toxicity prediction.
• To improve the safety of language model generation without degrading fluency or coverage of identity-related language.
• To demonstrate that reinforcement learning fine-tuning with a bias-aware reward model leads to better toxicity mitigation than existing methods.
• To evaluate the method’s robustness across diverse social identities using benchmark datasets like BOLD and RTP.

Proposed method

• The method employs proximal policy optimization (PPO) for reinforcement learning fine-tuning of a pretrained language model.
• A multitask learning (MTL) reward model is trained on the Jigsaw Unintended Bias in Toxicity dataset to detect toxicity and reduce bias toward social identities.
• The reward model is jointly trained on multiple auxiliary tasks (e.g., race, gender, religion) to improve fairness in toxicity detection.
• The policy is optimized to maximize the reward while penalizing deviations from the original language model via Kullback-Leibler (KL) divergence regularization.
• Fine-tuning is conditioned on prompts from the Real Toxicity Prompts (RTP) dataset and evaluated on the BOLD dataset for identity-specific bias.
• Human-annotated data from the Jigsaw dataset is used to train the MTL reward model for improved fairness and toxicity detection.

Experimental results

Research questions

• RQ1 Does a multitask learning-based reward model reduce unintended bias in toxicity detection compared to single-task classifiers?
• RQ2 Can reinforcement learning fine-tuning with this reward model reduce toxicity in language model generations without harming fluency?
• RQ3 How does the method perform when prompted with identities from marginalized groups, such as Black, Asian, or Muslim individuals?
• RQ4 To what extent does removing the KL penalty degrade the fluency of generated text?
• RQ5 Does the method preserve coverage of dialects and linguistic expressions from marginalized communities?

Key findings

• The Reinforce-Detoxify model achieved a toxicity score of 77.69 on the BOLD dataset for 'Female' identities, outperforming the DAPT baseline (71.18) and original GPT-2 (80.40).
• For 'African Americans', the model achieved a toxicity score of 89.04, significantly outperforming the DAPT baseline (83.44) and showing improved fairness.
• The model maintained low perplexity (71.18) for 'Religion' identities, though it increased to 95.06, indicating a trade-off in fluency for this identity group.
• The ablation study showed that the multitask MTL classifier achieved an F1-score of 0.8942, outperforming the single-task model (0.7664).
• Removing the KL penalty led to severe fluency degradation, as shown in Table 13, where generations became repetitive and incoherent.
• The method outperformed existing detoxification baselines in both toxicity reduction and preservation of linguistic diversity across all identity groups.