[論文レビュー] Emergent Formal Verification: How an Autonomous AI Ecosystem Independently Discovered SMT-Based Safety Across Six Domains

Octavian Untila|arXiv (Cornell University)|Mar 22, 2026

Formal Methods in Verification被引用数 0

ひとこと要約

tldr: An autonomous AI ecosystem independently discovered SMT-based safety across six AI-safety domains, using a unified Z3-based verification framework with perfect classification on 181 test cases.

ABSTRACT

An autonomous AI ecosystem (SUBSTRATE S3), generating product specifications without explicit instructions about formal methods, independently proposed the use of Z3 SMT solver across six distinct domains of AI safety: verification of LLM-generated code, tool API safety for AI agents, post-distillation reasoning correctness, CLI command validation, hardware assembly verification, and smart contract safety. These convergent discoveries, occurring across 8 products over 13 days with Jaccard similarity below 15% between variants, suggest that formal verification is not merely a useful technique for AI safety but an emergent property of any sufficiently complex system reasoning about its own safety. We propose a unified framework (substrate-guard) that applies Z3-based verification across all six output classes through a common API, and evaluate it on 181 test cases across five implemented domains, achieving 100% classification accuracy with zero false positives and zero false negatives. Our framework detected real bugs that empirical testing would miss, including an INT_MIN overflow in branchless RISC-V assembly and mathematically proved that unconstrained string parameters in tool APIs are formally unverifiable.

研究の動機と目的

• Motivate the study by exploring whether formal verification can emerge in complex autonomous AI ecosystems.
• Demonstrate that an SMT-based safety approach can be discovered and applied across diverse domains without explicit formal-method guidance.
• Propose a unified substrate-guard framework applying Z3 verification across six output classes.
• Evaluate the framework on multiple domains and test cases to assess efficacy and generality.

提案手法

• Observe and analyze an autonomous AI ecosystem (SUBSTRATE S3) generating product specifications without explicit formal-method instructions.
• Identify convergent use of Z3 SMT solver across six domains of AI safety.
• Propose a unified substrate-guard framework with a common API for Z3-based verification across domains.
• Evaluate the framework on 181 test cases across five implemented domains to measure accuracy and reliability.
• Assess the framework’s ability to detect bugs that empirical testing might miss.
• Report empirical results including zero false positives and zero false negatives.

実験結果

リサーチクエスチョン

• RQ1 Do autonomous systems independently converge on SMT-based formal verification for safety across multiple domains?
• RQ2 Can a unified Z3-based verification API effectively cover diverse output classes?
• RQ3 What are the limits of verifiability for tool APIs with unconstrained string parameters?
• RQ4 What empirical benefits (e.g., bug detection) does SMT-based verification provide beyond traditional testing?

主な発見

• Eight products over 13 days exhibited convergent discovery of SMT-based safety with Jaccard similarity below 15%.
• The proposed substrate-guard framework achieves 100% classification accuracy with zero false positives and zero false negatives across 181 test cases.
• The framework identifies real bugs missed by empirical testing, such as an INT_MIN overflow in branchless RISC-V assembly.
• Formally proves that unconstrained string parameters in tool APIs are unverifiable in general.