[Paper Review] Learning and Leveraging Verifiers to Improve Planning Capabilities of Pre-trained Language Models

Daman Arora, Subbarao Kambhampati|arXiv (Cornell University)|May 26, 2023

Topic Modeling4 citations

TL;DR

This paper proposes a verifier-augmented planning framework for fine-tuned LLMs, where a separate verifier model is trained to detect invalid actions in plan trajectories. By using randomly sampled negative examples from the same dataset, the verifier prunes illegal actions, improving success rates by 27% over baseline generators and reducing bad transitions from 52% to 5% in Blocksworld planning tasks.

ABSTRACT

There have been wide spread claims in the literature about the emergent reasoning capabilities of Pretrained Large Language Models. However, recent studies, have found that their ability to plan remains questionable. Through our experiments using GPT-2, we empirically demonstrate that the performance of a finetuned baseline remains poor because it violates pre-conditions of actions in the plans that it generates. To improve the planning capabilities of a finetuned LLM, we train a verifier, which can classify actions as being valid or invalid in a particular state. By randomly sampling actions from the same dataset, we generate examples of invalid actions which are then used to train a verifier which can check for action applicability. In the presence of diverse sampling from a generator and a verifier which can prune invalid trajectories, we show significant gains in the success rate on the Blocksworld domain. Additionally, we show that finetuning the GPT-2 generator itself to create the verifier generalizes better than finetuning the base GPT-2. Lastly, we investigate the role of the sampling temperature which can be used to control the exploration-exploitation tradeoff.

Motivation & Objective

• To address the poor planning performance of fine-tuned LLMs, which often violate action pre-conditions.
• To investigate whether a learned verifier can improve plan quality by detecting invalid actions in real-time.
• To explore whether finetuning the generator itself for verification generalizes better than using a base LLM.
• To analyze the impact of sampling temperature on the exploration-exploitation tradeoff in plan generation.
• To evaluate whether verifier-augmented generation scales better with increased generation attempts than baseline models.

Proposed method

• Train a verifier model on a dataset of successful trajectories by generating negative examples via random action sampling from the same dataset.
• Use the verifier to classify whether a given action is applicable in a given state, based on pre-conditions.
• Integrate the verifier into a generate-and-test loop: the generator samples multiple trajectories, and the verifier filters out invalid actions.
• Fine-tune the generator as the verifier (V_generator) instead of using a base GPT-2 model (V_base) to improve generalization.
• Control exploration-exploitation via sampling temperature during generation, with hyperparameter search over τ ∈ {0.2, 0.5, 0.8, 1.1}.
• Evaluate performance using goal-reaching rate (GRR) and bad transition rate (BTR) across varying numbers of generation attempts (k ∈ {1, 8, 16, 25}).

Figure 1: The figure shows a prompt which contains $s_{i}$ and $s_{g}$ , which is given to the generator as input and the output it generates, which contains $a_{i}$ and $s_{i+1}$ . The verifier taken the current state $s_{i}$ and generated action $a_{i}$ as input. After each transition generation,

Experimental results

Research questions

• RQ1 Can a self-supervised verifier trained on negative examples from the same dataset significantly improve the success rate of LLM-based planners?
• RQ2 Does finetuning the generator model itself to serve as a verifier lead to better generalization than finetuning a base LLM?
• RQ3 How does sampling temperature affect the balance between exploration and exploitation in plan generation?
• RQ4 Does the verifier-augmented generator scale better with increasing generation attempts than the baseline generator?
• RQ5 To what extent does the verifier reduce the rate of invalid actions in generated trajectories?

Key findings

• The generator with verifier achieves a 27% higher goal-reaching rate (GRR) compared to the baseline generator when given 25 generation attempts.
• The rate of bad transitions (invalid actions) drops from 52% in the baseline to just 5% when using the verifier.
• The performance of the baseline generator plateaus after 16 attempts, while the verifier-augmented model continues to improve with more attempts.
• Finetuning the generator itself to create the verifier (V_generator) yields better results (GRR = 0.655) than finetuning the base GPT-2 (V_base, GRR = 0.635).
• Sampling temperature has a non-monotonic effect on performance, with an optimal value around τ = 0.8, indicating a trade-off between exploration and exploitation.
• The verifier-based approach enables consistent scaling with more generation attempts, demonstrating that verification is critical for reliable LLM planning.

Figure 2: GRR of generator+verifier@k and generator@k with increasing number of attempts at generating a plan