[Paper Review] Skyline Computation with Noisy Comparisons

Benoît Groz, Frederik Mallmann-Trenn+2 more|arXiv (Cornell University)|Oct 5, 2017

Data Management and Algorithms25 references3 citations

TL;DR

This paper presents two novel output-sensitive algorithms for skyline computation under noisy comparisons, achieving query complexities of $O(nd\log(dk/\delta))$ and $O(ndk\log(k/\delta))$, respectively. The key contribution is closing the gap between known lower bounds and upper bounds in low-dimensional settings, resolving an open question on whether skyline computation can be done in $o(nk)$ queries for constant $d$, and proving the first tight bounds in this regime.

ABSTRACT

Given a set of $n$ points in a $d$-dimensional space, we seek to compute the skyline, i.e., those points that are not strictly dominated by any other point, using few comparisons between elements. We adopt the noisy comparison model [FRPU94] where comparisons fail with constant probability and confidence can be increased through independent repetitions of a comparison. In this model motivated by Crowdsourcing applications, Groz & Milo [GM15] show three bounds on the query complexity for the skyline problem. We improve significantly on that state of the art and provide two output-sensitive algorithms computing the skyline with respective query complexity $O(nd\\log (dk/\\delta))$ and $O(ndk\\log (k/\\delta))$ where $k$ is the size of the skyline and $\\delta$ the expected probability that our algorithm fails to return the correct answer. These results are tight for low dimensions.

Motivation & Objective

• To address the challenge of computing skylines efficiently when comparisons between points are subject to noise, as in crowdsourced data collection.
• To reduce the number of comparison queries required to compute the skyline with high probability, especially when the skyline size $k$ is small relative to $n$.
• To close the gap between known lower bounds and prior upper bounds in the noisy comparison model, particularly for low-dimensional data.
• To establish tight query complexity bounds for skyline computation, showing that $O(nd\log(dk/\delta))$ is optimal when $d \leq k^c$ for constant $c$.
• To develop a new subroutine for evaluating disjunctions of noisy boolean variables, which is instrumental in the main algorithms.

Proposed method

• The first algorithm, SkyLowDim, uses randomized sampling to assign points to buckets based on coordinate quantiles, reducing the input size while preserving skyline integrity with high probability.
• It employs a recursive bucketing strategy with super-exponentially increasing estimates of skyline size $\hat{k}$ to avoid costly retries, ensuring overall query complexity is bounded.
• The second algorithm, SkyHighDim, uses a different approach based on iterative elimination of dominated points, leveraging noisy comparisons to maintain confidence in dominance relations.
• A core subroutine, NoisyFirstTrue, is developed to identify the first true variable in a noisy boolean disjunction, which is used to efficiently test dominance across dimensions.
• The algorithms are designed to be trust-preserving, meaning they eliminate dependence on $\delta$ when the error probability is already bounded below $1/3$, improving practical efficiency.
• Theoretical analysis combines probabilistic concentration bounds and recursive cost decomposition to bound the total number of queries and running time.

Experimental results

Research questions

• RQ1 Can the skyline be computed in $o(nk)$ queries for constant dimension $d$, despite noisy comparisons?
• RQ2 What is the optimal query complexity for skyline computation in the noisy comparison model, particularly when the skyline size $k$ is small?
• RQ3 Is it possible to achieve output-sensitive complexity that depends on $k$ rather than $n$, even under noisy comparisons?
• RQ4 Can the query complexity be made independent of $\delta$ when the error probability is already bounded away from 1/2?
• RQ5 What is the information-theoretic lower bound on the number of queries required to compute the skyline with bounded error in the noisy model?

Key findings

• The paper establishes a tight query complexity of $\Omega(nd\log k)$ for skyline computation in the noisy comparison model when $d \leq k^c$ for some constant $c$, proving this is optimal.
• The proposed algorithm SkyLowDim achieves $O(nd\log(dk/\delta))$ query complexity, which is optimal for low-dimensional settings and improves upon the prior state of the art by a factor of $k$.
• The algorithm SkyHighDim achieves $O(ndk\log(k/\delta))$ query complexity, further improving on previous bounds and providing a second output-sensitive approach.
• The paper resolves an open question from [18] by showing that skyline computation is possible in $o(nk)$ queries for constant $d$, confirming that $O(ndk\log(k/\delta))$ is achievable and optimal.
• The authors show that SkyLowDim can be adapted to the noiseless setting, achieving $O(nd\log(dk))$ query complexity, which is optimal for that model.