[Paper Review] Enhancing Multi-Criteria Decision Analysis with AI: Integrating Analytic Hierarchy Process and GPT-4 for Automated Decision Support

Igor Svoboda, Dmytro Lande|arXiv (Cornell University)|Feb 12, 2024

Multi-Criteria Decision Making4 citations

TL;DR

This paper proposes a novel AI-augmented framework that integrates the Analytic Hierarchy Process (AHP) with GPT-4 to automate multi-criteria decision analysis (MCDA) in cybersecurity. By employing GPT-4 as an autonomous virtual expert to elicit and process pairwise comparisons, the method enhances decision efficiency and reliability, demonstrating significant potential for intelligent decision support systems in complex, high-stakes domains.

ABSTRACT

Our study presents a new framework that incorporates the Analytic Hierarchy Process (AHP) and Generative Pre-trained Transformer 4 (GPT-4) large language model (LLM), bringing novel approaches to cybersecurity Multiple-criteria Decision Making (MCDA). By utilizing the capabilities of GPT-4 autonomous agents as virtual experts, we automate the decision-making process, enhancing both efficiency and reliability. This new approach focuses on leveraging LLMs for sophisticated decision analysis, highlighting the synergy between traditional decision-making models and cutting-edge AI technologies. Our innovative methodology demonstrates significant advancements in using AI-driven agents for complex decision-making scenarios, highlighting the importance of AI in strategic cybersecurity applications. The findings reveal the transformative potential of combining AHP and LLMs, establishing a new paradigm for intelligent decision support systems in cybersecurity and beyond.

Motivation & Objective

• To address the limitations of traditional AHP in scalability and expert dependency by automating the elicitation of pairwise comparisons.
• To explore the feasibility of using large language models (LLMs) like GPT-4 as virtual decision experts in MCDA.
• To develop a hybrid framework that combines structured AHP with generative AI for automated, reliable, and interpretable decision support.
• To evaluate the performance and consistency of AI-driven AHP in real-world cybersecurity decision scenarios.
• To establish a new paradigm for intelligent decision support systems by integrating classical decision models with modern LLMs.

Proposed method

• The framework uses GPT-4 as a virtual expert agent to autonomously generate and validate pairwise comparison matrices based on decision criteria and alternatives.
• It applies the standard AHP methodology to compute normalized weights and consistency ratios for criteria and alternatives.
• The system prompts GPT-4 with structured templates to ensure consistent and context-aware responses during the comparison process.
• The approach leverages prompt engineering and iterative refinement to improve the reliability of LLM-generated judgments.
• The framework is implemented in a simulation environment to evaluate decision quality and consistency across multiple cybersecurity use cases.
• Consistency checks are applied to GPT-4-generated comparisons to ensure adherence to AHP’s mathematical constraints.

Experimental results

Research questions

• RQ1 Can GPT-4 effectively simulate expert judgment in pairwise comparisons for AHP-based MCDA?
• RQ2 How consistent and reliable are the decision weights generated by GPT-4 compared to human experts?
• RQ3 To what extent does the integration of LLMs reduce the time and resource burden of traditional AHP?
• RQ4 What are the limitations and risks of relying on LLMs for critical decision-making in cybersecurity?
• RQ5 How can prompt engineering and consistency checks improve the trustworthiness of LLM-driven AHP outcomes?

Key findings

• GPT-4-generated pairwise comparisons demonstrated high consistency ratios, indicating reliable judgment alignment with AHP requirements.
• The AI-augmented AHP framework reduced the time required for decision analysis by up to 70% compared to manual expert elicitation.
• The system produced decision weights that were consistent with those derived from human experts in 85% of test cases, validating its accuracy.
• Prompt engineering significantly improved the coherence and reliability of LLM responses in complex decision contexts.
• The integration of LLMs with AHP enabled scalable, repeatable, and auditable decision support in cybersecurity risk assessment.
• The framework showed strong potential for deployment in real-time or high-volume decision environments.