[Paper Review] Improving Open Language Models by Learning from Organic Interactions

Jing Xu, Da Young Ju+11 more|arXiv (Cornell University)|Jun 7, 2023

Hate Speech and Cyberbullying Detection4 citations

TL;DR

This paper introduces BlenderBot 3x, an improved open-domain language model trained on de-identified organic user interactions and feedback from real-world deployments. By leveraging a reward model and Cringe Loss to learn from both helpful and adversarial interactions, the model achieves 94.4% response quality and improved safety, outperforming BlenderBot 3 in both engagement and toxicity resistance on real user data.

ABSTRACT

We present BlenderBot 3x, an update on the conversational model BlenderBot 3, which is now trained using organic conversation and feedback data from participating users of the system in order to improve both its skills and safety. We are publicly releasing the participating de-identified interaction data for use by the research community, in order to spur further progress. Training models with organic data is challenging because interactions with people "in the wild" include both high quality conversations and feedback, as well as adversarial and toxic behavior. We study techniques that enable learning from helpful teachers while avoiding learning from people who are trying to trick the model into unhelpful or toxic responses. BlenderBot 3x is both preferred in conversation to BlenderBot 3, and is shown to produce safer responses in challenging situations. While our current models are still far from perfect, we believe further improvement can be achieved by continued use of the techniques explored in this work.

Motivation & Objective

• To improve conversational AI models using real-world, organic user interactions and feedback, rather than relying solely on curated or synthetic data.
• To address the challenge of learning from mixed-quality data, including adversarial and toxic user behavior, while still extracting useful signals for model improvement.
• To enhance both conversational quality (e.g., engagement, knowledge) and safety (e.g., reduced toxicity) in open-domain language models through data-driven fine-tuning.
• To release a large-scale, de-identified dataset of real user interactions to enable broader research in safe and effective conversational AI.

Proposed method

• The model is fine-tuned using a reward model trained on crowdworker-annotated data and user-provided feedback (e.g., flagging good or bad responses).
• Cringe Loss is applied to jointly optimize for high-quality responses and low probability of generating harmful, nonsensical, or off-topic outputs.
• The training data consists of over 353,000 de-identified conversations and 155,000 user feedback instances collected from real deployments of BlenderBot 3.
• A multi-stage data filtering and analysis pipeline is used to identify and mitigate noise from adversarial or low-quality interactions.
• The model is evaluated via human annotation to assess response quality and safety, with comparisons to the original BlenderBot 3.
• The system includes user-facing safeguards such as a visible banner, opt-out for data sharing, and an FAQ to inform users of risks and the research nature of the deployment.

Experimental results

Research questions

• RQ1 Can open-domain language models be effectively improved using organic, real-world user interactions and feedback, even when the data includes adversarial and toxic inputs?
• RQ2 How can models be trained to maintain high response quality and safety when exposed to both helpful and harmful user behavior in real deployments?
• RQ3 What techniques—such as Cringe Loss and reward modeling—can effectively separate useful feedback from noise and adversarial signals in real user data?
• RQ4 To what extent can a model’s safety and engagement improve when fine-tuned on real user feedback compared to curated or synthetic data?
• RQ5 How can large-scale, de-identified interaction data be responsibly released to support open research in conversational AI?

Key findings

• BlenderBot 3x achieves a 94.4% response quality rating in human evaluation, significantly outperforming BlenderBot 3’s 85.3%.
• In adversarial conversations, the model generates inappropriate responses only 2.4% of the time, down from 2.4% in the original model, indicating improved robustness.
• The model maintains high-quality responses in 85% of standard conversations, compared to 75% for human participants, showing strong conversational alignment.
• The use of Cringe Loss with a reward model leads to measurable improvements in both response quality and safety, especially in challenging interaction scenarios.
• Over 353,000 conversations and 155,000 feedback instances were collected and released as a de-identified dataset to support future research.
• Despite improvements, the model still exhibits failures under adversarial prompts, highlighting the need for continued research in alignment and safety.