[Paper Review] APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning

Yang Gao, Christian M. Meyer+1 more|arXiv (Cornell University)|Aug 29, 2018

Topic Modeling27 references6 citations

TL;DR

APRIL proposes a novel interactive learning framework that combines active preference learning and reinforcement learning to perform extractive multi-document summarisation without reference summaries. By separating the process into an active preference learning phase and a reinforcement learning phase, APRIL reduces sample complexity and significantly outperforms existing methods in both simulation and human evaluation, with users consistently preferring APRIL-generated summaries over those from SPPI.

ABSTRACT

We propose a method to perform automatic document summarisation without using reference summaries. Instead, our method interactively learns from users' preferences. The merit of preference-based interactive summarisation is that preferences are easier for users to provide than reference summaries. Existing preference-based interactive learning methods suffer from high sample complexity, i.e. they need to interact with the oracle for many rounds in order to converge. In this work, we propose a new objective function, which enables us to leverage active learning, preference learning and reinforcement learning techniques in order to reduce the sample complexity. Both simulation and real-user experiments suggest that our method significantly advances the state of the art. Our source code is freely available at https://github.com/UKPLab/emnlp2018-april.

Motivation & Objective

• To address the high sample complexity of preference-based interactive learning in NLP, especially for document summarisation.
• To eliminate reliance on expensive reference summaries by learning directly from user preferences.
• To improve the efficiency and robustness of interactive summarisation through a novel objective function that separates active preference learning and reinforcement learning phases.
• To evaluate the framework in both simulated and real-user settings, demonstrating its superiority over existing methods like SPPI.
• To enable practical deployment of interactive summarisation systems with minimal human interaction.

Proposed method

• APRIL introduces a new objective function that splits the learning process into two phases: Active Preference Learning (APL) and Reinforcement Learning (RL).
• In the APL phase, the system actively selects the most informative sentence pairs for user comparison, minimizing the number of required preference queries.
• The RL phase uses the collected preferences as sparse rewards to train a summarisation policy, optimizing for summary quality under length constraints.
• The framework supports integration of various APL and RL techniques, enabling flexible and efficient learning.
• APRIL uses a ranking-based reward mechanism instead of ROUGE scores, making it compatible with neural reinforcement learning models.
• The system is evaluated using both simulated oracles with varying noise levels and real human users in controlled comparison experiments.

Experimental results

Research questions

• RQ1 Can a preference-based interactive learning framework reduce sample complexity in extractive multi-document summarisation without reference summaries?
• RQ2 How does separating active preference learning from reinforcement learning improve learning efficiency and robustness?
• RQ3 Can APRIL outperform the state-of-the-art SPPI framework in both automated metrics and human evaluation?
• RQ4 How does APRIL perform under noisy or imperfect user feedback conditions?
• RQ5 To what extent can real users prefer APRIL-generated summaries over those from existing interactive systems?

Key findings

• APRIL achieved significantly higher summary quality than SPPI in simulation experiments, even with high-noise oracles, using only 10 rounds of interaction.
• In human evaluation, all seven users preferred the summaries generated by APRIL over those from SPPI across three distinct topics from DUC datasets.
• APRIL-generated summaries received consistently higher ratings on a 5-point Likert scale, confirming their superior quality in real-user settings.
• The framework demonstrated strong noise robustness, maintaining high performance even when user feedback contained errors or inconsistencies.
• The separation of APL and RL phases enabled more effective query selection and better reward utilization, reducing the number of required interactions.
• APRIL outperformed the ROUGE-2 upper bound of .212 on DUC’04, significantly narrowing the gap to the theoretical maximum.