[Paper Review] Probabilistic Software Modeling: A Data-driven Paradigm for Software Analysis

Hannes Thaller, Lukas Linsbauer+2 more|arXiv (Cornell University)|Dec 17, 2019

Software Testing and Debugging Techniques58 references4 citations

TL;DR

This paper introduces Probabilistic Software Modeling (PSM), a data-driven paradigm that analyzes traditional software via static and dynamic analysis to synthesize probabilistic models capable of simulating and quantifying program behavior. The key contribution is a scalable, reusable framework enabling predictive and generative software engineering applications such as test case generation, anomaly detection, and causal inference, validated across multiple real-world projects with strong empirical results.

ABSTRACT

Software systems are complex, and behavioral comprehension with the increasing amount of AI components challenges traditional testing and maintenance strategies.The lack of tools and methodologies for behavioral software comprehension leaves developers to testing and debugging that work in the boundaries of known scenarios. We present Probabilistic Software Modeling (PSM), a data-driven modeling paradigm for predictive and generative methods in software engineering. PSM analyzes a program and synthesizes a network of probabilistic models that can simulate and quantify the original program's behavior. The approach extracts the type, executable, and property structure of a program and copies its topology. Each model is then optimized towards the observed runtime leading to a network that reflects the system's structure and behavior. The resulting network allows for the full spectrum of statistical inferential analysis with which rich predictive and generative applications can be built. Applications range from the visualization of states, inferential queries, test case generation, and anomaly detection up to the stochastic execution of the modeled system. In this work, we present the modeling methodologies, an empirical study of the runtime behavior of software systems, and a comprehensive study on PSM modeled systems. Results indicate that PSM is a solid foundation for structural and behavioral software comprehension applications.

Motivation & Objective

• To address the growing challenge of behavioral comprehension in complex software systems, especially with increasing AI component integration.
• To develop a methodology that enables predictive and generative software engineering applications without requiring changes to programming languages or development workflows.
• To bridge the gap between traditional software engineering and probabilistic modeling by synthesizing statistical models from existing codebases.
• To evaluate the viability of probabilistic models in capturing structural and behavioral semantics of real-world software systems.

Proposed method

• Extract program structure via static code analysis, focusing on types, methods, fields, and properties while abstracting away control flow and statements.
• Collect dynamic runtime behavior by observing property accesses and method invocations during execution.
• Construct a network of probabilistic models by combining static structure and observed runtime data, using low-capacity models to avoid overfitting.
• Optimize models using likelihood-based training, with conditional probability tables (CPDs) for discrete variables and normalizing flows for continuous ones.
• Enable statistical inference through sampling, conditioning, and likelihood evaluation to support applications like test generation and anomaly detection.
• Use a prototype tool, Gradient, to implement and evaluate the approach on real-world Java projects.

Experimental results

Research questions

• RQ1 Can probabilistic models be effectively synthesized from traditional object-oriented software using static and dynamic analysis?
• RQ2 To what extent can these models simulate and quantify the behavioral characteristics of real software systems?
• RQ3 How well do the models generalize across different software projects of varying size and architecture?
• RQ4 Can the models support practical software engineering applications such as test case generation and anomaly detection?
• RQ5 What are the limitations of the approach in handling complex or highly dynamic code structures?

Key findings

• The empirical evaluation shows that PSM models achieve low and consistent negative log-likelihood (NLL) across diverse projects, indicating strong model fit and generalization.
• Most models have low dimensionality, supporting the use of low-capacity models without overfitting, as training and test NLL values are not significantly different.
• Qualitative inspection reveals high-quality model approximations, though issues like mass leakage and mode connectivity occur due to suboptimal model capacity.
• The inference pipeline successfully supports multi-dimensional information propagation and causal reasoning, enabling consistent generation and conditioning across models.
• The approach naturally integrates with AI components, enabling unified analysis of both traditional and AI-enhanced software systems.
• The prototype implementation demonstrates that PSM models are repeatable, persistable, shareable, and quantifiable, forming a solid foundation for downstream applications.