[论文解读] KGCE: Knowledge-Augmented Dual-Graph Evaluator for Cross-Platform Educational Agent Benchmarking with Multimodal Language Models

Zixian Liu, Sihao Liu+1 more|arXiv (Cornell University)|Jan 4, 2026

Multimodal Machine Learning Applications被引用 0

一句话总结

KGCE 引入一个知识库增强的双图评估框架，用于跨平台教育代理，可在 Windows 与 Android 任务以及私有领域软件中实现任务完成度与执行效率的细粒度度量。

ABSTRACT

With the rapid adoption of multimodal large language models (MLMs) in autonomous agents, cross-platform task execution capabilities in educational settings have garnered significant attention. However, existing benchmark frameworks still exhibit notable deficiencies in supporting cross-platform tasks in educational contexts, especially when dealing with school-specific software (such as XiaoYa Intelligent Assistant, HuaShi XiaZi, etc.), where the efficiency of agents often significantly decreases due to a lack of understanding of the structural specifics of these private-domain software. Additionally, current evaluation methods heavily rely on coarse-grained metrics like goal orientation or trajectory matching, making it challenging to capture the detailed execution and efficiency of agents in complex tasks. To address these issues, we propose KGCE (Knowledge-Augmented Dual-Graph Evaluator for Cross-Platform Educational Agent Benchmarking with Multimodal Language Models), a novel benchmarking platform that integrates knowledge base enhancement and a dual-graph evaluation framework. We first constructed a dataset comprising 104 education-related tasks, covering Windows, Android, and cross-platform collaborative tasks. KGCE introduces a dual-graph evaluation framework that decomposes tasks into multiple sub-goals and verifies their completion status, providing fine-grained evaluation metrics. To overcome the execution bottlenecks of existing agents in private-domain tasks, we developed an enhanced agent system incorporating a knowledge base specific to school-specific software. The code can be found at https://github.com/Kinginlife/KGCE.

研究动机与目标

• 解决缺乏教育性、跨平台任务基准与私有域软件支持的问题。
• 引入知识库以提升面向学校专用应用的 MLM 驱动代理执行效果。
• 提出双图评估器（任务完成图和执行效率图）用于细粒度任务评估。
• 创建一个涵盖 Windows、Android 以及跨平台协作的 104 个教育相关任务数据集。

提出的方法

• 构建一个涵盖 Windows、Android 以及跨平台协作的 104 任务教育数据集，组织为 DAG 子任务图。
• 构建一个学校专用的 JSON 知识库，包含软件包、页面和 UI 元素，用于动态增强提示。
• 引入 Knowledge Invocation Decision，在相关时将 KB 数据注入提示。
• 为细粒度评估定义 Task Completeness Graph（CR、CPA）和 Execution Efficiency Graph（BR、Precision、Recall、F1、OoR、RMS）。
• 在 MLMs（Qwen-VL-Max-Latest、GPT-4o、Gemini-2.0-Flash）上进行有无 KB 的对比评估，以衡量性能提升。

Figure 1: The overall framework of KGCE. The system first generates tasks from educational datasets, then executes them through a pipeline of action prediction, execution, and evaluation. A dual-graph evaluator assesses task completeness and execution efficiency. Based on screenshot and OCR feedback

实验结果

研究问题

• RQ1 RQ1：双图评估框架是否在粗略指标之外提供有意义的细粒度洞察？
• RQ2 RQ2：知识库增强如何影响不同任务与模型的代理表现？
• RQ3 RQ3：不同多模态语言模型在有无知识库支持的 KGCE 任务上表现如何？
• RQ4 RQ4：哪种模型从构建的知识库中获得的收益最大？

主要发现

指标	无 KB (%)	有 KB (%)	提升 (%)
CR	60.02	75.26	+25.39
CPA	7.22	11.29	+56.37
Precision	24.68	32.84	+33.06
Recall	63.87	75.79	+18.66
F1-score	33.96	44.96	+32.39
BR	52.01	41.47	-20.27
OoR	13.42	7.54	-43.81
RMS	46.33	31.27	-32.51

• 知识库增强显著提升所有关键指标在各模型上的表现，例如 CR 从 60.02% 提升至 75.26%，CPA 从 7.22% 提升至 11.29%。
• 有 KB 时执行效率提升：RMS 从 46.33 降至 31.27，OoR 从 13.42 降至 7.54。
• GPT-4o 总体表现最佳，带有 KB 时 CR 达到 77.21%，F1 为 47.71%。
• Qwen-VL-Max-Latest 在 CR 上的相对 KB 驱动提升最大（从 52.88% 提升到 76.53%）。
• Gemini-2.0-Flash 也从 KB 获益（CR 61.80% 提升至 72.03%），但 OoR 略有恶化，可能与规则与动态推理之间的冲突有关。

Figure 2: The knowledge base module. Given a task description and package names, the system identifies relevant packages, retrieves their descriptions from the knowledge base, and constructs prompts for the LLM. The knowledge base is organized by packages, pages, and elements, with each element incl