[Paper Review] EventRL: Enhancing Event Extraction with Outcome Supervision for Large Language Models

Jun Gao, Huan Zhao+3 more|arXiv (Cornell University)|Feb 18, 2024

Topic Modeling4 citations

TL;DR

EventRL proposes a reinforcement learning framework that enhances large language models (LLMs) for event extraction by using outcome supervision with task-specific reward functions to reduce hallucinations and structural errors. It outperforms few-shot prompting and supervised fine-tuning, especially in identifying novel event types and maintaining correct event structure, with CodeLLaMa models enhanced by code data showing superior generalization and performance gains up to 65.90 F1-score on held-in data.

ABSTRACT

In this study, we present EventRL, a reinforcement learning approach developed to enhance event extraction for large language models (LLMs). EventRL utilizes outcome supervision with specific reward functions to tackle prevalent challenges in LLMs, such as instruction following and hallucination, manifested as the mismatch of event structure and the generation of undefined event types. We evaluate EventRL against existing methods like Few-Shot Prompting (FSP) (based on GPT4) and Supervised Fine-Tuning (SFT) across various LLMs, including GPT-4, LLaMa, and CodeLLaMa models. Our findings show that EventRL significantly outperforms these conventional approaches by improving the performance in identifying and structuring events, particularly in handling novel event types. The study emphasizes the critical role of reward function selection and demonstrates the benefits of incorporating code data for better event extraction. While increasing model size leads to higher accuracy, maintaining the ability to generalize is essential to avoid overfitting.

Motivation & Objective

• To address instruction following and hallucination issues in LLM-based event extraction, particularly structural mismatches and undefined event types.
• To improve event extraction performance by shifting from standard loss functions to outcome-based feedback that reflects actual extraction accuracy.
• To investigate whether outcome supervision via reinforcement learning can enhance generalization to unseen event types and improve structural fidelity in event schemas.
• To evaluate the impact of code data pretraining on event extraction performance and model robustness across different model scales.

Proposed method

• EventRL employs a reinforcement learning framework where the policy network is optimized using outcome-based reward functions derived from event extraction performance metrics.
• Three reward functions are introduced: Argument-F1, Average-F1, and Product-F1, each designed to penalize incorrect argument roles or event type predictions more severely.
• The method incorporates Teacher-Force Threshold and Advantage Clipping to stabilize training and prevent policy degradation and catastrophic forgetting.
• Reward shaping is applied based on exact match of event arguments and types, with feedback directly tied to the correctness of the final event structure.
• The approach is applied to LLMs including GPT-4, LLaMA, CodeLLaMa, and others, with training using SFT-initialized models fine-tuned via PPO-like optimization.
• Code data pretraining is integrated to enhance the model’s ability to understand structured information, improving performance on event extraction tasks.

Figure 1: Examples of common errors in LLM-Based event extraction. The left side depicts an error of generating an undefined event type, specifically an unexpected “Vote” event not included in the guidelines. The right side shows a structural mismatch error within an “Attack” event, incorporating an

Experimental results

Research questions

• RQ1 Can outcome supervision via reinforcement learning significantly reduce hallucination and structural errors in LLM-based event extraction?
• RQ2 How do different reward functions (Argument-F1, Average-F1, Product-F1) affect the model’s ability to learn correct event structures and types?
• RQ3 Does pretraining on code data improve the generalization of LLMs to unseen event types in event extraction?
• RQ4 To what extent does increasing model size improve event extraction performance, and does it lead to overfitting on held-out data?
• RQ5 Can outcome-based feedback via EventRL outperform standard supervised fine-tuning and few-shot prompting in zero-shot and few-shot event extraction settings?

Key findings

• EventRL significantly outperforms both few-shot prompting (GPT-4) and supervised fine-tuning (SFT) across multiple LLMs, with a 65.90 F1-score on the held-in test set using CodeLLaMa-13B and the Product-F1 reward function.
• CodeLLaMa models pre-trained on code data achieve 59.23 AVG F1 on held-in and 49.74 on held-out tests at 7B scale, outperforming LLaMa-7B (56.03 and 37.35), demonstrating the benefit of code data for structured reasoning.
• The performance gap between CodeLLaMa and LLaMa widens at 13B scale, with CodeLLaMa-13B achieving 51.69 AVG F1 on held-out data versus LLaMa’s 42.04.
• When scaled to 34B parameters, CodeLLaMa-34B shows a drop in generalization (48.75 F1 on held-out data), indicating overfitting risks with larger models.
• Case studies show EventRL (Prod-F1) correctly identifies the agent role ('police') and 'crime' argument in an arrest event, while SFT baseline misses both, highlighting improved structural accuracy.
• The method demonstrates that reward function design is critical: Product-F1 yields the best performance, suggesting that joint optimization of precision and recall is most effective for event structure fidelity.

Figure 2: The EventRL framework architecture, demonstrating the process from initialization with an SFT Model $M_{0}$ , through iterative model updates $M_{t}$ to $M_{t+1}$ via Outcome Supervision. This includes using reinforcement learning with reward functions based on Trigger-F1 and Argument-F1 s