[Paper Review] Shepherd: A Critic for Language Model Generation

Tianlu Wang, Ping Yu+8 more|arXiv (Cornell University)|Aug 8, 2023

Topic Modeling4 citations

TL;DR

Shepherd is a 7B-parameter language model fine-tuned to critique and refine outputs from other large language models, using a high-quality feedback dataset combining community forum critiques and human-annotated feedback. It achieves performance on par with GPT-4 and outperforms models like Alpaca and SelFee in both automatic and human evaluations, demonstrating strong capability in identifying factual, logical, and coherence errors with actionable suggestions.

ABSTRACT

As large language models improve, there is increasing interest in techniques that leverage these models' capabilities to refine their own outputs. In this work, we introduce Shepherd, a language model specifically tuned to critique responses and suggest refinements, extending beyond the capabilities of an untuned model to identify diverse errors and provide suggestions to remedy them. At the core of our approach is a high quality feedback dataset, which we curate from community feedback and human annotations. Even though Shepherd is small (7B parameters), its critiques are either equivalent or preferred to those from established models including ChatGPT. Using GPT-4 for evaluation, Shepherd reaches an average win-rate of 53-87% compared to competitive alternatives. In human evaluation, Shepherd strictly outperforms other models and on average closely ties with ChatGPT.

Motivation & Objective

• To develop a dedicated critique model that enhances the reliability and quality of LLM-generated text by identifying factual, logical, and coherence errors.
• To create a high-quality, diverse feedback dataset combining community-sourced critiques and human-annotated feedback for training.
• To evaluate the effectiveness of a small, specialized critic model against larger, general-purpose models like ChatGPT in generating actionable, accurate feedback.
• To enable iterative refinement of LLM outputs through automated, high-quality critique, reducing hallucinations and improving factual consistency.

Proposed method

• Fine-tune a 7B-parameter language model on a curated feedback dataset composed of community feedback from Stack Exchange and human-annotated critiques on model generations.
• Use a two-stage data collection process: first, extract critiques from real online Q&A forums to capture diverse, informal feedback; second, collect high-quality, structured feedback from human annotators on model outputs across diverse tasks.
• Train Shepherd to generate natural language feedback that identifies specific errors (e.g., factual inaccuracies, logical flaws) and provides actionable suggestions for improvement.
• Evaluate critique quality using both GPT-4 as a judge in automated preference comparisons and human annotators in pairwise preference studies.
• Compare Shepherd’s feedback against baselines including Alpaca, SelFee, and ChatGPT across seven diverse datasets to assess consistency, accuracy, and helpfulness.
• Analyze the relative contributions of community vs. human-annotated data, finding community data enhances diversity while human data improves precision and quality.

Figure 2: Preference evaluation using GPT-4 as the evaluator, comparing Shepherd with 3 different competing models across 7 different datasets.

Experimental results

Research questions

• RQ1 Can a small, specialized language model outperform general-purpose LLMs in generating high-quality, actionable critiques for LLM-generated text?
• RQ2 How does the inclusion of diverse, informal community feedback compare to curated human-annotated feedback in improving critique model performance?
• RQ3 To what extent can a critic model like Shepherd reduce hallucinations and improve factual consistency in LLM outputs?
• RQ4 How does Shepherd’s feedback quality compare to that of GPT-4 and ChatGPT in automated and human evaluations?

Key findings

• Shepherd achieves a win-rate of 53–87% over competitive baselines in GPT-4-based preference evaluation, indicating strong preference for its critiques across diverse datasets.
• In human evaluation, Shepherd strictly outperforms Alpaca and SelFee and closely ties with ChatGPT, demonstrating high-quality, actionable feedback.
• Community feedback data contributes greater diversity and coverage of error types, while human-annotated data improves precision and reduces ambiguity in critiques.
• Shepherd identifies specific errors such as factual inaccuracies (e.g., confusing baking soda with baking powder) and logical inconsistencies (e.g., claiming 'no-risk' investments exist) more effectively than baselines.
• Alpaca frequently gives uniformly positive feedback regardless of output quality, while SelFee often provides vague or off-topic feedback, highlighting the need for specialized critique models.
• The model’s 7B parameter size enables strong performance without requiring large-scale inference, making it a lightweight yet effective tool for iterative refinement of LLM outputs.

Figure 3: Human preference evaluation, comparing Shepherd with 3 different competing models across 7 different datasets.