[Paper Review] Concurrency Control for Adaptive Indexing

Goetz Graefe, Felix Halim+3 more|arXiv (Cornell University)|Mar 29, 2012

Advanced Database Systems and Queries23 references4 citations

TL;DR

This paper proposes a concurrency control mechanism for adaptive indexing that treats index structure updates as optional, short-term in-memory operations, avoiding traditional transactional locks. By decoupling index structure refinement from content updates and exploiting dynamic workload adaptation, it achieves negligible performance overhead even under concurrent queries, with concurrency conflicts decreasing over time as indexing evolves.

ABSTRACT

Adaptive indexing initializes and optimizes indexes incrementally, as a side effect of query processing. The goal is to achieve the benefits of indexes while hiding or minimizing the costs of index creation. However, index-optimizing side effects seem to turn read-only queries into update transactions that might, for example, create lock contention. This paper studies concurrency control in the context of adaptive indexing. We show that the design and implementation of adaptive indexing rigorously separates index structures from index contents; this relaxes the constraints and requirements during adaptive indexing compared to those of traditional index updates. Our design adapts to the fact that an adaptive index is refined continuously, and exploits any concurrency opportunities in a dynamic way. A detailed experimental analysis demonstrates that (a) adaptive indexing maintains its adaptive properties even when running concurrent queries, (b) adaptive indexing can exploit the opportunity for parallelism due to concurrent queries, (c) the number of concurrency conflicts and any concurrency administration overheads follow an adaptive behavior, decreasing as the workload evolves and adapting to the workload needs.

Motivation & Objective

• Address the challenge of concurrency overhead in adaptive indexing, where read-only queries trigger index structure updates.
• Reduce or eliminate the performance penalties typically associated with locking and transactional overhead during index refinement.
• Enable scalable, high-concurrency execution of read-only queries by treating index structure changes as optional and short-lived.
• Demonstrate that concurrency control overhead adapts to workload dynamics, decreasing as indexing matures.
• Explore dynamic, workload-aware strategies for choosing adaptive indexing algorithms to further improve concurrency and performance.

Proposed method

• Decouple index structure updates from logical index content, ensuring only physical design changes occur during query processing.
• Use in-memory latches instead of long-term transactional locks for structural updates, minimizing contention and overhead.
• Implement opportunistic and incremental index refinement that can be skipped or delayed based on workload patterns.
• Dynamically adapt locking granularity as indexing progresses, enabling finer-grained concurrency control over time.
• Introduce 'active' and 'lazy' strategies for index refinement, with dynamic algorithm selection based on query workload and concurrency levels.
• Leverage multiple concurrent queries to guide unified, bulk index refinement operations, increasing parallelism and reducing per-query overhead.

Experimental results

Research questions

• RQ1 Can concurrency control overhead in adaptive indexing be reduced to negligible levels despite read-only queries triggering structural updates?
• RQ2 How does the dynamic nature of adaptive indexing affect concurrency conflicts and locking requirements compared to traditional indexing?
• RQ3 To what extent can concurrent queries be exploited to improve parallelism and reduce structural update overhead?
• RQ4 What are the performance implications of using 'active' versus 'lazy' strategies for index refinement under concurrent workloads?
• RQ5 Can workload-aware, dynamic selection of adaptive indexing algorithms further reduce concurrency overhead and improve scalability?

Key findings

• Adaptive indexing maintains its adaptive behavior even under concurrent query workloads, with no degradation in performance or refinement quality.
• Concurrency conflicts decrease over time as the index structure evolves and locking granularity improves, leading to reduced contention.
• The system dynamically adapts to workload characteristics, with concurrency control overheads shrinking as indexing progresses and becomes more stable.
• Concurrent queries can be leveraged to trigger bulk, unified index refinement operations, increasing parallelism and reducing per-operation costs.
• Using in-memory latches instead of transactional locks for structural updates results in negligible performance overhead, even under high concurrency.
• The experimental evaluation confirms that adaptive early termination and dynamic algorithm selection further alleviate concurrency control challenges in practice.