[Paper Review] Approaching Human-Level Forecasting with Language Models

Danny Halawi, Fred Zhang+2 more|arXiv (Cornell University)|Feb 28, 2024

demographic modeling and climate adaptation5 citations

TL;DR

A retrieval-augmented language-model system is built to forecast binary events, achieving near-crowd performance and, in some settings, surpassing it, by retrieving up-to-date news, reasoning with scratchpad prompts, and ensembling forecasts.

ABSTRACT

Forecasting future events is important for policy and decision making. In this work, we study whether language models (LMs) can forecast at the level of competitive human forecasters. Towards this goal, we develop a retrieval-augmented LM system designed to automatically search for relevant information, generate forecasts, and aggregate predictions. To facilitate our study, we collect a large dataset of questions from competitive forecasting platforms. Under a test set published after the knowledge cut-offs of our LMs, we evaluate the end-to-end performance of our system against the aggregates of human forecasts. On average, the system nears the crowd aggregate of competitive forecasters, and in some settings surpasses it. Our work suggests that using LMs to forecast the future could provide accurate predictions at scale and help to inform institutional decision making.

Motivation & Objective

• Motivate automated forecasting to match human forecasters in binary-event prediction.
• Leverage retrieval-augmented prompting to incorporate up-to-date information.
• Develop a self-supervised fine-tuning approach to improve forecast reasoning.
• Evaluate end-to-end forecasting against crowd aggregates on a recent, large dataset.

Proposed method

• Build a retrieval-augmented LM pipeline with query generation, relevance ranking, and article summarization.
• Use a reasoning module that generates forecasts from question context and article summaries via carefully designed scratchpad prompts.
• Fine-tune a model on self-supervised data where model outputs outperform the crowd to improve reasoning.
• Ensemble multiple forecasts using a trimmed mean to produce a final prediction.
• Evaluate end-to-end system with Brier score and calibration metrics against crowd aggregates.
• Hyperparameter sweep to optimize retrieval, prompting, and ensembling strategies.

(a) Our retrieval system . The LM takes in the question and generates search queries to retrieve articles from historical news APIs. Then the LM ranks the articles on relevancy and summarizes the top $k$ articles.

Experimental results

Research questions

• RQ1 Can a retrieval-augmented LM system forecast binary events at or near human crowd performance?
• RQ2 How do retrieval quality, reasoning prompts, and ensembling affect forecast accuracy and calibration?
• RQ3 Does self-supervised fine-tuning of reasoning improve forecasting performance over zero-shot baselines?
• RQ4 Under what selective or selective-with-criteria settings does the system outperform human crowds?

Key findings

• The end-to-end system approaches crowd performance on all questions with a Brier score of 0.179 versus 0.149 for the crowd (average over the test set).
• In aggregated accuracy, the system achieves 71.5% versus 77.0% for the crowd, across all questions.
• Under selective settings and with sufficient relevant articles, the system can outperform the crowd aggregate on some metrics.
• The system remains well-calibrated, with RMS calibration comparable to the crowd and improvements over base models in zero-shot settings.
• Retrieval of at least 5 relevant articles and early retrieval dates improve performance relative to the crowd.
• Fine-tuning a reasoning LM on data where forecasts beat the crowd yields stronger forecasting capabilities.

(b) Our reasoning system . The system takes in the question and summarized articles and prompts LMs to generate forecasts. The forecasts are then aggregated into a final forecast using the trimmed mean.