[Paper Review] A Scalable Test Problem Generator for Sequential Transfer Optimization

Xiaoming Xue, Cuie Yang+4 more|arXiv (Cornell University)|Apr 17, 2023

Advanced Multi-Objective Optimization Algorithms4 citations

TL;DR

This paper proposes a scalable test problem generator for sequential transfer optimization (STO) that enables customizable similarity distributions between source and target tasks, addressing the lack of systematic benchmarking in STO research. By using inverse transform sampling to explicitly define similarity distributions, the generator creates diverse, realistic STO problems that expose performance differences among source selection algorithms, demonstrating superior evaluation capability over existing benchmarks.

ABSTRACT

Sequential transfer optimization (STO), which aims to improve the optimization performance on a task of interest by exploiting the knowledge captured from several previously-solved optimization tasks stored in a database, has been gaining increasing research attention over the years. However, despite the remarkable advances in algorithm design, the development of a systematic benchmark suite for comprehensive comparisons of STO algorithms received far less attention. Existing test problems are either simply generated by assembling other benchmark functions or extended from specific practical problems with limited scalability. The relationships between the optimal solutions of the source and target tasks in these problems are also often manually configured, limiting their ability to model different similarity relationships presented in real-world problems. Consequently, the good performance achieved by an algorithm on these problems might be biased and hard to be generalized to other problems. In light of the above, in this study, we first introduce four concepts for characterizing STO problems and present an important problem feature, namely similarity distribution, which quantitatively delineates the relationship between the optima of the source and target tasks. Then, we present the general design guidelines of STO problems and a particular STO problem generator with good scalability. Specifically, the similarity distribution of a problem can be easily customized, enabling a continuous spectrum of representation of the diverse similarity relationships of real-world problems. Lastly, a benchmark suite with 12 STO problems featured by a variety of customized similarity relationships is developed using the proposed generator. The source code of the problem generator is available at https://github.com/XmingHsueh/STOP-G.

Motivation & Objective

• To address the lack of systematic benchmark suites for evaluating sequential transfer optimization (STO) algorithms.
• To identify the critical role of similarity distribution between source and target tasks as a key problem feature in STO benchmarking.
• To develop a scalable, extensible test problem generator that supports customizable similarity distributions for diverse STO problem generation.
• To create a benchmark suite of 12 STO problems with varied similarity relationships to enable fair and comprehensive algorithm evaluation.
• To demonstrate the superiority of the proposed generator in distinguishing performance differences among source selection methods compared to existing synthetic test problems.

Proposed method

• Introduces four foundational concepts: task family, task-optimum mapping, optimum coverage, and similarity distribution to characterize STO problems.
• Proposes a novel optimum configuration scheme based on inverse transform sampling to explicitly control the similarity distribution between source and target task optima.
• Designs a general framework for STO problem generation that supports scalability and extensibility for diverse optimization scenarios.
• Employs elemental functions and their combinations to construct synthetic STO problems with tunable similarity relationships.
• Uses evolutionary algorithms with standard operators (simulated binary crossover, polynomial mutation) as the backbone optimizer in evaluation experiments.
• Employs Wilcoxon rank-sum tests to statistically compare algorithm rankings across different test problem sets.

Experimental results

Research questions

• RQ1 Can a test problem generator with customizable similarity distributions better differentiate the performance of source selection methods in STO than existing benchmarks?
• RQ2 How do varying similarity distributions between source and target tasks affect the ranking and relative performance of STO algorithms?
• RQ3 To what extent does the proposed generator support the creation of a diverse and representative benchmark suite for STO evaluation?
• RQ4 Can the proposed STO problems reveal performance differences among source selection algorithms that are masked by existing test problems?
• RQ5 How does the similarity distribution feature influence the effectiveness of knowledge transfer in STO algorithms?

Key findings

• The proposed test problem generator enables explicit, customizable specification of similarity distributions between source and target tasks, overcoming the limitations of fixed or manually configured relationships in prior work.
• Existing test problems exhibit limited similarity variation, with most source tasks showing either very high or very low similarity to the target, leading to tied rankings among source selection algorithms.
• In contrast, the proposed STO problems with mixed similarity distributions reveal distinct and statistically significant performance differences across source selection methods, demonstrating their ability to serve as a more effective evaluation arena.
• Algorithms like subspace alignment (SA), relaxed ordinal correlation (ROC), and Wasserstein distance (WD) show improved ranking and performance consistency on the new benchmark suite, indicating better sensitivity to similarity structure.
• The benchmark suite of 12 STO problems, generated using the proposed framework, successfully captures a broad spectrum of similarity relationships, enabling more comprehensive and reliable algorithm evaluation.
• The source code for the problem generator is publicly available at https://github.com/XmingHsueh/STOP-G, supporting reproducibility and future extension.