[Paper Review] Real-Time Web Scale Event Summarization Using Sequential Decision Making

Chris Kedzie, Fernando Díaz+1 more|arXiv (Cornell University)|May 12, 2016

Advanced Text Analysis Techniques20 references16 citations

TL;DR

This paper presents a real-time, web-scale streaming summarization system that uses sequential decision making via 'learning to search' to dynamically select relevant, comprehensive, and timely updates from live news streams. It achieves a 28.3% improvement in standard F₁ and a 43.8% improvement in time-sensitive F₁ over state-of-the-art baselines by jointly modeling relevance, novelty, comprehensiveness, and timeliness in an online setting.

ABSTRACT

We present a system based on sequential decision making for the online summarization of massive document streams, such as those found on the web. Given an event of interest (e.g. "Boston marathon bombing"), our system is able to filter the stream for relevance and produce a series of short text updates describing the event as it unfolds over time. Unlike previous work, our approach is able to jointly model the relevance, comprehensiveness, novelty, and timeliness required by time-sensitive queries. We demonstrate a 28.3% improvement in summary F1 and a 43.8% improvement in time-sensitive F1 metrics.

Motivation & Objective

• To address the challenge of summarizing massive, real-time document streams from the web, especially for time-sensitive events like breaking news or crises.
• To overcome limitations of traditional extractive summarization methods that assume batch processing and full access to documents.
• To jointly optimize for relevance, novelty, comprehensiveness, and timeliness in streaming summarization.
• To develop a fully online system capable of processing web-scale data with low latency and minimal redundancy.
• To improve upon existing streaming summarization systems that rely on batch processing or simplistic sentence selection heuristics.

Proposed method

• The system frames streaming summarization as a sequential decision-making problem, where each sentence is either selected or skipped in real time.
• It employs a 'learning to search' approach inspired by reinforcement learning, training a policy network to imitate an oracle summarizer.
• The model uses features such as sentence similarity, topic coverage, and temporal novelty to guide decisions.
• It incorporates a similarity filter and first-sentence prioritization to reduce redundancy and improve efficiency.
• The system is trained end-to-end on streaming data using a structured prediction objective that accounts for cumulative summarization quality.
• It evaluates and updates summaries incrementally, ensuring timeliness and avoiding redundant or irrelevant content.

Experimental results

Research questions

• RQ1 Can a sequential decision-making framework effectively balance relevance, novelty, comprehensiveness, and timeliness in real-time event summarization?
• RQ2 How does learning to search compare to traditional extractive summarization and batch processing methods in streaming settings?
• RQ3 To what extent does modeling the full document (beyond the first sentence) improve summary quality and reduce errors?
• RQ4 How do different error types—misses, false positives, duplicates—affect system performance, and can they be mitigated?
• RQ5 Does a policy-based approach that considers future summarization quality outperform heuristic or greedy selection strategies?

Key findings

• The proposed system achieved a 28.3% improvement in standard F₁ score over state-of-the-art baselines on a diverse set of real-world event queries.
• It demonstrated a 43.8% improvement in time-sensitive F₁, highlighting its effectiveness for latency-sensitive applications like crisis monitoring.
• The LsCos variant outperformed all other systems in both latency-penalized and unpenalized evaluations, with statistically significant gains.
• While Cos (first-sentence-only) had high error rates—especially 41.1% empty errors—Ls-based models reduced such false positives to just 1% when combined with similarity filtering.
• Missed content, particularly in article bodies, was the largest error category, accounting for up to 30% of errors, indicating a need for better content identification.
• The system produced summaries that were significantly more concise than Cos (400% fewer updates), making it more suitable for real-time monitoring applications.