[Paper Review] Scheduling Refresh Queries for Keeping Results from a SPARQL Endpoint Up-to-Date (Extended Version)

Magnus Knuth, Olaf Hartig+1 more|arXiv (Cornell University)|Aug 29, 2016

Data Quality and Management8 references3 citations

TL;DR

This paper proposes a middleware-based scheduling strategy for refreshing SPARQL query results on dynamic RDF datasets to prevent SPARQL endpoint overload. By analyzing real-world query dynamics on DBpedia Live over three months, it evaluates scheduling strategies based on historical change frequency and adaptive result lifetimes, finding that dynamic, history-aware scheduling minimizes missed changes and reduces time-to-detection with optimal resource use.

ABSTRACT

Many datasets change over time. As a consequence, long-running applications that cache and repeatedly use query results obtained from a SPARQL endpoint may resubmit the queries regularly to ensure up-to-dateness of the results. While this approach may be feasible if the number of such regular refresh queries is manageable, with an increasing number of applications adopting this approach, the SPARQL endpoint may become overloaded with such refresh queries. A more scalable approach would be to use a middle-ware component at which the applications register their queries and get notified with updated query results once the results have changed. Then, this middle-ware can schedule the repeated execution of the refresh queries without overloading the endpoint. In this paper, we study the problem of scheduling refresh queries for a large number of registered queries by assuming an overload-avoiding upper bound on the length of a regular time slot available for testing refresh queries. We investigate a variety of scheduling strategies and compare them experimentally in terms of time slots needed before they recognize changes and number of changes that they miss.

Motivation & Objective

• To address the scalability problem of frequent SPARQL query refreshes that can overload public endpoints.
• To design a middleware system that schedules query re-evaluations without overwhelming the SPARQL endpoint.
• To evaluate scheduling strategies under a hard time-slot constraint to avoid endpoint overload.
• To identify the most effective scheduling heuristics based on historical change patterns in real-world SPARQL queries.
• To minimize missed changes and reduce time-to-detection of result updates in dynamic RDF datasets.

Proposed method

• The approach uses a middleware component where applications register their SPARQL queries and receive notifications upon result changes.
• A time-slot constraint limits the total duration available per scheduling cycle to prevent endpoint overload.
• Scheduling strategies are evaluated based on historical change frequency (dynamics) and adaptive result lifetimes.
• Queries are re-executed only when the estimated time-to-change exceeds the available time slot, prioritizing high-dynamic queries.
• The system tracks result isomorphism and binding order (for ORDER BY queries) to detect actual changes.
• Experiments use 10,000 real-world SPARQL queries from LSQ on DBpedia Live over 3 months, with runtime and change detection metrics collected per revision.

Experimental results

Research questions

• RQ1 Which scheduling strategy minimizes the time to detect changes in SPARQL query results under a fixed time-slot budget?
• RQ2 How does the historical change frequency (dynamics) of query results influence scheduling performance?
• RQ3 Can adaptive result lifetimes improve detection accuracy and reduce missed changes?
• RQ4 What is the impact of query type (e.g., SELECT, ASK) and runtime on scheduling efficiency?
• RQ5 How do periods of high data update frequency affect the detectability of result changes?

Key findings

• The change history of query results is the most influential factor in determining scheduling effectiveness.
• Scheduling strategies based on previously recognized change frequency (dynamics) achieved the best detection performance with minimal missed changes.
• Adaptively allocated maximum lifetimes for individual query results significantly improved detection speed and reduced false negatives.
• Only 352 out of 10,000 queries changed across all revisions, with up to 32 changes per revision, indicating low overall update frequency.
• The average runtime for all queries per revision ranged from 440 to 870 seconds, with affected queries consuming up to 8.9% of total time.
• Periods of high data update frequency (e.g., late August 2015) correlated strongly with increased numbers of changed query results.