[Paper Review] CUBES: A Parallel Synthesizer for SQL Using Examples

Ricardo Brancas, Miguel Terra-Neves+3 more|arXiv (Cornell University)|Mar 9, 2022

Advanced Database Systems and Queries4 citations

TL;DR

CUBES is a parallel SQL synthesizer that uses input-output examples to generate queries, leveraging divide-and-conquer parallelism across multicore architectures and a fuzzing-based disambiguation procedure to improve accuracy. It achieves 60% accuracy in matching user intent—significantly higher than prior tools—while scaling with super-linear speedups on hard instances.

ABSTRACT

In recent years, more people have seen their work depend on data manipulation tasks. However, many of these users do not have the background in programming required to write complex programs, particularly SQL queries. One way of helping these users is automatically synthesizing the SQL query given a small set of examples. Several program synthesizers for SQL have been recently proposed, but they do not leverage multicore architectures. This paper proposes CUBES, a parallel program synthesizer for the domain of SQL queries using input-output examples. Since input-output examples are an under-specification of the desired SQL query, sometimes, the synthesized query does not match the user's intent. CUBES incorporates a new disambiguation procedure based on fuzzing techniques that interacts with the user and increases the confidence that the returned query matches the user intent. We perform an extensive evaluation on around 4000 SQL queries from different domains. Experimental results show that our sequential version can solve more instances than other state-of-the-art SQL synthesizers. Moreover, the parallel approach can scale up to 16 processes with super-linear speedups for many hard instances. Our disambiguation approach is critical to achieving an accuracy of around 60%, significantly larger than other SQL synthesizers.

Motivation & Objective

• Address the scalability limitations of existing SQL synthesizers that do not utilize multicore architectures.
• Improve the accuracy of synthesized queries by resolving ambiguity between multiple correct but semantically different queries that satisfy the same input-output examples.
• Enable interactive disambiguation with users via fuzzing-generated test cases to refine query selection and align with user intent.
• Extend the expressiveness and efficiency of SQL query synthesis through pruning techniques and domain-specific language (DSL) splitting in a parallel framework.
• Provide a publicly available benchmark of ~4000 SQL synthesis instances for future research evaluation.

Proposed method

• Adapt a sequential SQL synthesizer with enhanced query language support and pruning heuristics to reduce invalid program enumeration.
• Implement a divide-and-conquer parallelization strategy where each process explores a distinct subset of the DSL by assigning different operation sequences and DSL fragments.
• Use fuzzing to automatically generate new test cases (input tables) that produce different outputs for candidate queries, enabling targeted disambiguation.
• Interactively query the user on the correctness of outputs from these fuzzed test cases to eliminate incorrect candidate queries iteratively.
• Balance load and avoid starvation by dynamically increasing the number of DSL operations per thread, ensuring work distribution across processes.
• Integrate the disambiguation loop into the synthesis pipeline to refine results until user intent is confidently matched.

Experimental results

Research questions

• RQ1 Can a parallel query synthesizer based on input-output examples outperform sequential state-of-the-art tools in solving complex SQL synthesis problems?
• RQ2 To what extent can parallelism in multicore systems improve the performance and scalability of SQL query synthesis?
• RQ3 How effective is a fuzzing-based interactive disambiguation procedure in increasing the accuracy of synthesized queries beyond the baseline of 50%?
• RQ4 Can pruning techniques and DSL splitting significantly reduce the search space while maintaining expressiveness in SQL synthesis?
• RQ5 How does the accuracy of query synthesis tools compare when measured by alignment with user intent rather than just syntactic correctness?

Key findings

• The sequential version of CUBES solves more SQL synthesis instances than existing state-of-the-art tools, demonstrating improved completeness and robustness.
• The parallel version of CUBES achieves super-linear speedups on many hard instances, with a median speedup of 15× when using 16 processes.
• Without disambiguation, existing synthesizers achieve less than 50% accuracy in matching user intent, indicating a critical gap in practical usability.
• With the proposed fuzzing-based disambiguation, CUBES increases accuracy to approximately 60%, significantly outperforming prior tools.
• The framework successfully handles a diverse benchmark of ~4000 SQL queries across multiple domains, including extended natural language-based examples.
• The integration of interactive disambiguation via user feedback on fuzzed test cases enables reliable identification of the intended query despite ambiguity in input-output examples.