[Paper Review] LLM-augmented Preference Learning from Natural Language

Inwon Kang, Sikai Ruan+5 more|arXiv (Cornell University)|Oct 12, 2023

Topic Modeling4 citations

TL;DR

This paper investigates the use of large language models (LLMs) for preference learning from natural language, framing the classification task as a prompt-based inference problem. It demonstrates that LLMs—especially with few-shot prompting—outperform prior state-of-the-art models, particularly on longer, complex texts, without fine-tuning.

ABSTRACT

Finding preferences expressed in natural language is an important but challenging task. State-of-the-art(SotA) methods leverage transformer-based models such as BERT, RoBERTa, etc. and graph neural architectures such as graph attention networks. Since Large Language Models (LLMs) are equipped to deal with larger context lengths and have much larger model sizes than the transformer-based model, we investigate their ability to classify comparative text directly. This work aims to serve as a first step towards using LLMs for the CPC task. We design and conduct a set of experiments that format the classification task into an input prompt for the LLM and a methodology to get a fixed-format response that can be automatically evaluated. Comparing performances with existing methods, we see that pre-trained LLMs are able to outperform the previous SotA models with no fine-tuning involved. Our results show that the LLMs can consistently outperform the SotA when the target text is large -- i.e. composed of multiple sentences --, and are still comparable to the SotA performance in shorter text. We also find that few-shot learning yields better performance than zero-shot learning.

Motivation & Objective

• To evaluate whether large language models (LLMs) can effectively classify comparative preferences in natural language without fine-tuning.
• To investigate the impact of prompt engineering and few-shot vs. zero-shot learning on preference classification performance.
• To compare LLM-based approaches against existing state-of-the-art models on benchmark datasets with varying text complexity.
• To explore the feasibility of using LLMs as a scalable, zero-shot capable solution for preference learning in multi-agent decision-making systems.
• To identify limitations of current LLMs in handling nuanced or implicit preference expressions and propose directions for improvement.

Proposed method

• Formalized the preference classification task as a prompt-based inference problem, where input text is embedded into a natural language instruction for LLMs.
• Designed two effective prompting strategies: one using explicit instruction templates and another incorporating few-shot examples within the prompt.
• Evaluated four LLMs: GPT-4, GPT-3.5-Turbo, LLaMa-2-70B, and LLaMa-2-13B, across two benchmark datasets: Compsent-19 and College Confidential.
• Used automated evaluation via fixed-format response parsing to ensure consistent and reliable performance measurement across models.
• Implemented zero-shot and few-shot prompting configurations to compare generalization and few-shot in-context learning capabilities.
• Applied a structured evaluation pipeline to isolate model performance from prompt engineering artifacts, ensuring reproducibility and fairness in comparison.

Figure 1. Illustration of the retry prompt process. The incorrect output is appended to the original prompt, followed by a retry message to remind the rules.

Experimental results

Research questions

• RQ1 Can LLMs outperform existing state-of-the-art models in preference learning from natural language without fine-tuning?
• RQ2 How does few-shot prompting compare to zero-shot prompting in terms of preference classification accuracy across different LLMs?
• RQ3 Does model size (e.g., GPT-4 vs. LLaMa-2-13B) significantly affect performance on longer or more complex preference expressions?
• RQ4 Can LLMs effectively handle implicit or multi-sentence preference expressions that challenge traditional NLP models?
• RQ5 What role does prompt design play in unlocking the full potential of LLMs for preference classification tasks?

Key findings

• Pre-trained LLMs, particularly GPT-4 and LLaMa-2-70B, outperform previous state-of-the-art models on the College Confidential dataset, which contains longer and more complex texts.
• Few-shot prompting consistently yields better performance than zero-shot prompting across all evaluated LLMs, indicating that in-context learning improves preference detection accuracy.
• Even the smaller LLaMa-2-13B model achieves competitive performance, suggesting that LLMs can be effective even with moderate parameter counts when properly prompted.
• On shorter, single-sentence texts (e.g., Compsent-19), some prior SotA models still outperform LLMs, indicating that LLMs are not universally superior but excel in complexity.
• The largest model, GPT-4, achieves the highest performance, demonstrating that model scale enhances reasoning and preference understanding in nuanced contexts.
• Prompt engineering significantly influences performance, with structured, example-rich prompts yielding more reliable and accurate predictions than generic instructions.