[Paper Review] Managing Schema Evolution in NoSQL Data Stores

Stefanie Scherzinger, Meike Klettke+1 more|arXiv (Cornell University)|Aug 2, 2013

Advanced Database Systems and Queries12 references40 citations

TL;DR

This paper proposes a declarative schema evolution language for NoSQL data stores that enables systematic management of schema changes—such as adding, renaming, or moving properties—without modifying the underlying database. The approach treats NoSQL stores as black boxes, using a generic database programming language to implement schema operations safely and efficiently across diverse NoSQL systems, addressing the critical challenge of maintaining data consistency during agile, frequent releases.

ABSTRACT

NoSQL data stores are commonly schema-less, providing no means for globally defining or managing the schema. While this offers great flexibility in early stages of application development, developers soon can experience the heavy burden of dealing with increasingly heterogeneous data. This paper targets schema evolution for NoSQL data stores, the complex task of adapting and changing the implicit structure of the data stored. We discuss the recommendations of the developer community on handling schema changes, and introduce a simple, declarative schema evolution language. With our language, software developers and architects can systematically manage the evolution of their production data and perform typical schema maintenance tasks. We further provide a holistic NoSQL database programming language to define the semantics of our schema evolution language. Our solution does not require any modifications to the NoSQL data store, treating the data store as a black box. Thus, we want to address application developers that use NoSQL systems

Motivation & Objective

• To address the lack of systematic schema management tools in schema-less NoSQL data stores used in modern web applications.
• To support agile development practices where schema evolves rapidly with weekly or daily releases.
• To provide a declarative, portable language for defining schema evolution operations across various NoSQL databases.
• To decouple schema evolution from the NoSQL backend by treating the data store as a black box.
• To enable safe, consistent migration of production data during application evolution.

Proposed method

• Design a declarative schema evolution language that supports common operations: adding, deleting, renaming, copying, and moving properties across entities.
• Implement the schema evolution language using a generic NoSQL database programming language to ensure compatibility with multiple NoSQL systems.
• Define semantics for schema operations using a holistic programming language that captures the behavior of schema changes.
• Ensure backward compatibility and data consistency by enforcing deterministic migration semantics.
• Avoid requiring modifications to the NoSQL data store, enabling use in database-as-a-service environments.
• Leverage existing object mappers and application-level abstractions to map schema changes to data store operations.

Experimental results

Research questions

• RQ1 How can schema evolution be systematically managed in NoSQL data stores that lack built-in schema definition or management?
• RQ2 What operations are necessary and safe for evolving the implicit schema of NoSQL-stored data in production environments?
• RQ3 How can schema evolution be expressed declaratively and portably across different NoSQL databases without backend modifications?
• RQ4 What are the implications of eager versus lazy schema migration in NoSQL systems, and how can both be supported safely?
• RQ5 How can existing object mappers and application frameworks be integrated with a schema evolution language for NoSQL data stores?

Key findings

• The proposed schema evolution language enables developers to declaratively express common schema changes—such as adding, renaming, or moving properties—across NoSQL data stores.
• The language is implemented via a generic NoSQL database programming language, ensuring broad compatibility with various NoSQL systems without requiring backend modifications.
• Schema operations are executed in a deterministic, safe manner, avoiding non-deterministic migrations that could compromise data consistency.
• The approach treats NoSQL data stores as black boxes, making it suitable for database-as-a-service environments where direct configuration is not available.
• The solution supports both eager and potential future lazy migration strategies, offering flexibility for different release cycles and data evolution patterns.
• Existing NoSQL object mappers (e.g., Objectify, Morphia) provide limited support for schema evolution; this work fills the gap with a systematic, declarative approach.