[Paper Review] Should I Have Expressed a Different Intent? Counterfactual Generation for LLM-Based Autonomous Control

Amirmohammad Farzaneh, Salvatore D'Oro+1 more|arXiv (Cornell University)|Jan 27, 2026

Explainable Artificial Intelligence (XAI)0 citations

TL;DR

The paper introduces conformal counterfactual generation (CCG) to produce reliable counterfactual reports for LLM-driven agent–environment control, outperforming naive re-execution baselines and providing reliability guarantees.

ABSTRACT

Large language model (LLM)-powered agents can translate high-level user intents into plans and actions in an environment. Yet after observing an outcome, users may wonder: What if I had phrased my intent differently? We introduce a framework that enables such counterfactual reasoning in agentic LLM-driven control scenarios, while providing formal reliability guarantees. Our approach models the closed-loop interaction between a user, an LLM-based agent, and an environment as a structural causal model (SCM), and leverages test-time scaling to generate multiple candidate counterfactual outcomes via probabilistic abduction. Through an offline calibration phase, the proposed conformal counterfactual generation (CCG) yields sets of counterfactual outcomes that are guaranteed to contain the true counterfactual outcome with high probability. We showcase the performance of CCG on a wireless network control use case, demonstrating significant advantages compared to naive re-execution baselines.

Motivation & Objective

• Enable counterfactual reasoning about alternative user intents in LLM-based autonomous control workflows.
• Model the agent–environment interaction as a structural causal model to generate counterfactual reports.
• Provide reliability guarantees via conformal calibration for sets of counterfactual outcomes.
• Demonstrate fidelity and reliability improvements over naive re-execution baselines in a wireless network (5G) control scenario.

Proposed method

• Model the agent–environment system as a structural causal model (SCM) capturing X (prompt), A (action), Z (environment feedback), and Y (report).
• Use abduction to infer environment noise U_Z from the factual episode T=(X,A,Z,Y).
• Generate counterfactual actions under X′ by replaying the LLM with the same exogenous A- and Y-noise U_A, U_Y.
• Simulate counterfactual environment feedback Ẑ_{X′}(T) using inferred U_Z and counterfactual actions.
• Produce counterfactual reports Ŷ_{X′}(T) from the report-generating LLM conditioned on (X′,Â_{X′}(T),Ẑ_{X′}(T)).
• Apply conformal calibration (CCG) to create a reliable set C_λ(T,X′) of counterfactual reports with high-probability coverage guarantees.

Figure 1 : (a) Example of the agentic framework studied in this work: A network operator provides an intent as a prompt to an LLM-based agent. The agent translates this prompt into an action, which is used to configure the network. The network executes the action, and returns environment feedback to

Experimental results

Research questions

• RQ1 How to reliably generate counterfactual reports for LLM-based autonomous control when the environment responds to actions?
• RQ2 Can we provide statistical reliability guarantees for counterfactual reports in closed-loop agent–environment systems?
• RQ3 Does conformal calibration improve the usefulness and reliability of counterfactual reasoning compared to naive re-execution?
• RQ4 What is the effect of simulator fidelity on counterfactual accuracy and calibration efficiency?

Key findings

• CG (counterfactual generation) yields higher fidelity counterfactual KPI time series than IG or SIG across multiple metrics.
• In a 5G network control case study, CG achieves MAE and cross-correlation closer to true counterfactuals compared to IG and SIG for both throughput and delay.
• An LLM-based judge prefers CG-generated counterfactual reports over IG and SIG in 92 of 100 cases.
• CCG provides reliability guarantees: the generated set C_λ(T,X′) contains a good counterfactual with probability at least 1−ε, with calibration controlling miscoverage.
• CCG achieves fewer excess samples than fixed-budget baselines across tested budgets, improving sampling efficiency.

Figure 2 : An example of a factual episode with a true and a generated counterfactual report. Emphasis (bold fonts) added to highlight the main differences between factual and counterfactual intents.