[Paper Review] OR-Benchmark: An Open and Reconfigurable Digital Watermarking Benchmarking Framework

Hui Wang, Anthony T. S. Ho+1 more|arXiv (Cornell University)|May 31, 2015

Advanced Steganography and Watermarking Techniques21 references3 citations

TL;DR

This paper proposes OR-Benchmark, an open and highly reconfigurable digital watermarking benchmarking framework that supports diverse watermarking applications through public, general-purpose interfaces and user-extensible configurations. It enables fair, repeatable performance evaluation of watermarking algorithms by standardizing benchmarking workflows and aligning performance metrics across test images, demonstrated through case studies on image authentication and self-restoration schemes.

ABSTRACT

Benchmarking digital watermarking algorithms is not an easy task because different applications of digital watermarking often have very different sets of requirements and trade-offs between conflicting requirements. While there have been some general-purpose digital watermarking benchmarking systems available, they normally do not support complicated benchmarking tasks and cannot be easily reconfigured to work with different watermarking algorithms and testing conditions. In this paper, we propose OR-Benchmark, an open and highly reconfigurable general-purpose digital watermarking benchmarking framework, which has the following two key features: 1) all the interfaces are public and general enough to support all watermarking applications and benchmarking tasks we can think of; 2) end users can easily extend the functionalities and freely configure what watermarking algorithms are tested, what system components are used, how the benchmarking process runs, and what results should be produced. We implemented a prototype of this framework as a MATLAB software package and used it to benchmark a number of digital watermarking algorithms involving two types of watermarks for content authentication and self-restoration purposes. The benchmarking results demonstrated the advantages of the proposed benchmarking framework, and also gave us some useful insights about existing image authentication and self-restoration watermarking algorithms which are an important but less studied topic in digital watermarking.

Motivation & Objective

• To address the lack of flexible, extensible, and standardized benchmarking environments for digital watermarking algorithms that can handle diverse application-specific requirements and conflicting performance trade-offs.
• To enable researchers to fairly compare multiple watermarking schemes under identical testing conditions, minimizing reliance on re-implementing prior evaluations.
• To support complex benchmarking tasks involving multiple watermarking types, system components, and performance evaluation (PE) metrics, including alignment of non-uniform performance data across test images.
• To provide a reusable, extensible framework that facilitates performance comparison and insight extraction for emerging watermarking applications such as content authentication and self-restoration.
• To promote reproducibility and community collaboration by releasing a MATLAB-based prototype under an open-source license with extensible functional units.

Proposed method

• Designing a modular benchmarking framework with public, general-purpose interfaces to support all known digital watermarking schemes, including those with multiple watermark types.
• Implementing a prototype in MATLAB that allows end users to freely configure tested algorithms, system components, benchmarking processes, and output results.
• Introducing a method to align performance indicators (e.g., PSNR, SSIM) across different test images by extending their domain to (−∞, ∞) and using a sign-based averaging function to compute average performance curves.
• Defining an average performance function: \bar{f}(x) = \frac{\sum_{i=1}^{N}f_{i}(x)}{\sum_{i=1}^{N}\text{sign}(f_{i}(x))} , where \text{sign}(x)=0 for x\leq 0 and 1 otherwise, to handle non-aligned performance data from images with varying bpp ranges.
• Using piecewise linear interpolation to estimate continuous bpp-PSNR and other IQA metric functions for each test image, enabling smooth averaging across all images.
• Extending the Performance Evaluation (PE) library with new algorithms to support benchmarking under varying compression levels (e.g., JPEG) and control factors like bpp, enabling consistent comparison across different settings.

Experimental results

Research questions

• RQ1 How can a general-purpose digital watermarking benchmarking framework be designed to support diverse applications with conflicting requirements and reconfigurable workflows?
• RQ2 What techniques can ensure fair and reproducible performance evaluation when benchmarking watermarking algorithms with non-uniform performance data across test images?
• RQ3 How can performance metrics like PSNR and SSIM be aligned across images with different dynamic ranges of control parameters (e.g., bpp) to enable meaningful averaging and comparison?
• RQ4 What impact do benchmarking implementation details—such as metric alignment and curve averaging—have on the final performance ranking of watermarking schemes?
• RQ5 How can a benchmarking framework be made extensible and community-driven to support ongoing research and reduce duplication of effort?

Key findings

• The OR-Benchmark framework successfully supports all known types of digital watermarking schemes, including complex ones with multiple watermark types, through its open and reconfigurable design.
• The proposed method for aligning performance indicators using domain extension and sign-based averaging enables consistent and fair comparison of watermarking schemes across test images with varying parameter ranges.
• Case studies on image authentication and self-restoration watermarking schemes revealed that benchmarking details significantly affect performance rankings, with M2 outperforming M1 at higher bpp values when using aligned metrics.
• The framework demonstrated that M3 remains the best-performing scheme overall, but performance rankings can shift depending on how metrics are processed and averaged, highlighting the importance of standardized benchmarking procedures.
• The prototype implementation in MATLAB enables researchers to easily extend functionality, configure testing workflows, and reuse results, reducing the need for re-implementing prior evaluations.
• The authors confirmed that the framework supports flexible benchmarking across different media types, attacks, datasets, and PE algorithms, with plans to expand functional units and release the code under an open-source license.