[Paper Review] Scalability Analysis of the RADAR Decision Support Tool

Saheed A. Busari, Emmanuel Letier|arXiv (Cornell University)|Feb 9, 2017

Simulation Techniques and Applications4 references3 citations

TL;DR

This paper analyzes the scalability of the RADAR decision support tool, which evaluates architecture decisions using multi-objective optimization and information value analysis. The study confirms linear growth in time and memory usage with respect to simulations, objectives, and design space size, identifying exhaustive simulation as the dominant performance bottleneck, limiting practical use to problems with up to ~10 independent decisions with 3 options each.

ABSTRACT

This report presents a theoretical complexity analysis and empirical scalability analysis of the Requirements and Architecture Decision Analyser (RADAR).

Motivation & Objective

• To evaluate the theoretical and empirical scalability of the RADAR decision support tool for architecture decision analysis.
• To identify performance bottlenecks in RADAR’s four-step analysis pipeline: design space generation, simulation, Pareto optimization, and information value computation.
• To determine the practical limits of RADAR’s exhaustive search strategy on real-world and synthetic models.
• To guide future development by identifying the need for heuristic search strategies to handle larger design spaces.
• To assess the impact of model characteristics—number of simulations, objectives, and design space size—on runtime and memory consumption.

Proposed method

• Conducts theoretical time and space complexity analysis of RADAR’s four-step pipeline using asymptotic notation.
• Empirically evaluates performance on synthetic models with controlled variations in number of simulations (N), design space size, and objectives.
• Uses a synthetic model generator to create models with configurable decisions, options, objectives, and dependencies.
• Measures runtime and memory usage across 180 synthetic models with varying design space sizes and up to 5 objectives.
• Applies standard formulas for EVTPI and EVPPI computation, with complexity bounded by O(N×|S|) for both.
• Uses a recent efficient algorithm for EVPPI estimation, maintaining linear time complexity with respect to the number of Pareto-optimal solutions.

Experimental results

Research questions

• RQ1 How does RADAR’s runtime and memory usage scale with the number of simulations (N)?
• RQ2 How does RADAR’s performance scale with increasing design space size?
• RQ3 How does RADAR’s performance scale with the number of objectives?
• RQ4 What proportion of time and memory is consumed by each of the four analysis steps?

Key findings

• RADAR’s runtime and memory usage increase linearly with the number of simulations (N), confirming theoretical complexity predictions.
• The simulation step consumes 100% of total runtime and 96% of memory usage, making it the dominant performance bottleneck.
• With 2 objectives and 10 decisions (3 options each), RADAR analyzed a design space of 6912 solutions in 111 seconds, under two minutes.
• The largest model analyzed in under an hour had a design space of 153,751 solutions (11 decisions with 7 options each), confirming scalability within practical limits.
• Runtime and memory usage scale linearly with the number of objectives, as predicted by theory, with no significant deviation observed.
• The Pareto optimization and expected information value computation steps contribute negligibly to runtime (0% and 0% respectively) and memory (1% and 3%), confirming simulation as the primary cost.