[Paper Review] Metamorphic Testing: A New Approach for Generating Next Test Cases

Tsong Yueh Chen, Shing-Chi Cheung+1 more|arXiv (Cornell University)|Feb 28, 2020

Software Testing and Debugging TechniquesComputer Science461 citations

TL;DR

Introduces metamorphic testing to derive new test cases from successful ones to uncover hidden software errors and operate without complete test oracles.

ABSTRACT

In software testing, a set of test cases is constructed according to some predefined selection criteria. The software is then examined against these test cases. Three interesting observations have been made on the current artifacts of software testing. Firstly, an error-revealing test case is considered useful while a successful test case which does not reveal software errors is usually not further investigated. Whether these successful test cases still contain useful information for revealing software errors has not been properly studied. Secondly, no matter how extensive the testing has been conducted in the development phase, errors may still exist in the software [5]. These errors, if left undetected, may eventually cause damage to the production system. The study of techniques for uncovering software errors in the production phase is seldom addressed in the literature. Thirdly, as indicated by Weyuker in [6], the availability of test oracles is pragmatically unattainable in most situations. However, the availability of test oracles is generally assumed in conventional software testing techniques. In this paper, we propose a novel test case selection technique that derives new test cases from the successful ones. The selection aims at revealing software errors that are possibly left undetected in successful test cases which may be generated using some existing strategies. As such, the proposed technique augments the effectiveness of existing test selection strategies. The technique also helps uncover software errors in the production phase and can be used in the absence of test oracles.

Motivation & Objective

• Motivate the limitations of traditional test case selection and the value of reusing successful test cases.
• Propose metamorphic testing as a technique to derive new test cases aimed at exposing undetected errors.
• Show how metamorphic testing augments existing test selection strategies and supports production-phase error detection.
• Address the absence of reliable test oracles in many practical scenarios.

Proposed method

• Describe metamorphic testing as deriving new test cases from successful ones.
• Explain how metamorphic relations guide the generation of follow-up test cases.
• Argue that this approach augments existing test strategies and can operate when test oracles are unavailable.
• Position metamorphic testing as a means to reveal errors potentially missed by conventional testing.

Experimental results

Research questions

• RQ1 How can new test cases be generated from successful ones to reveal hidden errors?
• RQ2 Can metamorphic testing augment traditional test selection strategies?
• RQ3 How does metamorphic testing help in production-phase error detection when test oracles are absent?

Key findings

• Metamorphic testing derives additional test cases from successful tests to uncover errors not revealed by existing strategies.
• The technique complements and enhances the effectiveness of existing test selection approaches.
• It provides a practical approach for error detection in the production phase where test oracles are often unavailable.