[Paper Review] When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels

Weiyan Shi, Emily Dinan+3 more|arXiv (Cornell University)|Oct 28, 2022

Topic Modeling4 citations

TL;DR

This paper proposes Juicer, a framework that converts sparse binary feedback and free-form textual feedback from real-world human-bot interactions into high-quality training data by training a satisfaction classifier and a reply corrector to generate improved responses. By augmenting dialogue models with these model-generated corrections, Juicer achieves performance comparable to models fine-tuned on 100% gold feedback, especially when combined with the Director model for improved response quality.

ABSTRACT

Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However, humans may not always provide explicit signals when the chatbot makes mistakes during interactions. In this work, we propose Juicer, a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a satisfaction classifier to label the unlabeled data; and (ii) training a reply corrector to map the bad replies to good ones. We find that augmenting training with model-corrected replies improves the final dialogue model, and we can further improve performance by using both positive and negative replies through the recently proposed Director model.

Motivation & Objective

• To address the challenge of sparse feedback in deployed dialogue systems, where users rarely provide binary thumbs up/down or gold corrections.
• To leverage dense free-form textual feedback—commonly provided when users express dissatisfaction—as a signal to improve model training.
• To develop a framework that automatically generates high-quality corrections for bad bot responses using feedback, enabling data augmentation.
• To improve dialogue model performance using only organic, real-world feedback signals collected during deployment.
• To evaluate whether model-generated corrections can match or exceed performance of models trained on full gold-annotated data.

Proposed method

• Train a binary satisfaction classifier on existing binary feedback to predict whether a bot response is satisfactory or not.
• Train a reply corrector model to generate improved responses conditioned on bad bot outputs and free-form textual feedback explaining the issues.
• Use the trained satisfaction classifier to label unlabeled bot responses in the dataset, expanding the scope of feedback.
• Generate corrected responses for low-satisfaction outputs using the reply corrector, creating synthetic high-quality training examples.
• Augment the original training data with the generated corrections and retrain the final dialogue model.
• Apply the Director model to further improve generation by jointly optimizing for both positive and negative response signals.

Experimental results

Research questions

• RQ1 Can free-form textual feedback be effectively leveraged to train a satisfaction classifier and reply corrector for dialogue systems?
• RQ2 Does augmenting training data with model-generated corrections improve dialogue model performance compared to using only gold corrections?
• RQ3 Can the combination of Juicer and the Director model further enhance response quality beyond either method alone?
• RQ4 How does the performance of Juicer compare to models trained on 100% gold-annotated feedback, which serves as an upper bound?
• RQ5 To what extent does the quality of the satisfaction classifier impact downstream performance in the feedback conversion pipeline?

Key findings

• Free-form textual feedback significantly improves the performance of both the satisfaction classifier and reply corrector, enabling effective identification and correction of bad responses.
• Augmenting training data with model-generated corrections leads to better F1 scores (18.5 vs. 18.0) on valid and unseen test sets compared to using only gold corrections.
• The Director model further improves performance, achieving 45.5% good responses in human evaluation, outperforming Juicer alone (41.9%).
• The best Juicer model achieves comparable results to models trained on 100% gold data: 17.7 F1 (vs. 17.6) and 45.5% good responses (vs. 47.0%) in human evaluation.
• Using a threshold to select only the most confident corrections improves performance, avoiding degradation from low-quality predictions.
• The framework outperforms baseline models even when the baseline uses 100% gold labels, demonstrating effective signal extraction from sparse feedback.