[论文解读] The Arrival of AGI? When Expert Personas Exceed Expert Benchmarks

Drake Mullens, Stella Shen|arXiv (Cornell University)|Mar 4, 2026

Persona Design and Applications被引用 0

一句话总结

论文重新审视专家人设是否会提升语言模型性能的空结果，识别导致空结果的结构性原因，并通过受控试验展示在测量限制被消除时，专家人设在有效项上可达到满分精度。

ABSTRACT

Do expert personas improve language model performance? The Wharton Generative AI Lab reports that they do not, broadcasting to millions via social media the recommendation that practitioners abandon a technique recommended by Anthropic, Google, and OpenAI. We demonstrate that this null finding was structurally predictable. Five core mechanisms precluded detection before data collection began: baseline contamination elevating the starting point to near-ceiling, system prompt hierarchy subordinating experimental manipulation, impossible expert specifications collapsing to generic competence, format constraints suppressing reasoning processes, and provider exclusion limiting generalizability. Controlled trials correcting these limitations reveal what the original design obscured. To test this, we selected the GPQA Diamond hardest questions to prevent baseline pattern matching, forcing reliance on genuine expert reasoning. On items with valid key answers, expert personas achieve ceiling accuracy. They eliminated all baseline errors through confidence amplification. Furthermore, forensic examination of model divergence identified that half of the hardest GPQA items contain chemically or logically indefensible answers. The model's CoT revealed reasoning away from impossible answers, yielding penalization for accurate chemistry. These findings recontextualize the original null results. Methodologically sound persona research faces measurement constraints imposed by benchmark validity limitations. Answering the persona question requires evaluation infrastructure the field does not yet possess.

研究动机与目标

• 评估专家人设是否改善语言模型性能。
• 识别在基准测试中模糊化人设效应的方法学约束。
• 通过受控试验展示在恰当评估下真正的专家推理如何显现。

提出的方法

• 识别并评析人设基准测试中的测量约束。
• 将 GPQA Diamond 最难问题用于减轻基线模式匹配。
• 通过受控试验校正基线污染、系统提示效应及其他偏差。
• 分析模型的思维过程 (CoT) 以理解推理和惩罚模式。
• 进行模型分歧的法证性检验以探测不可辩护的答案。

实验结果

研究问题

• RQ1 当以鲁棒基准测试评估时，专家人设是否改善语言模型性能？
• RQ2 哪些测量约束阻碍在标准基准中检测到人设效应？
• RQ3 在何种条件下专家人设能在困难问题上达到真正的专家级表现？

主要发现

• 最初的空结果是结构性可预见的，因为存在多种预先存在的偏见。
• 在 GPQA Diamond 最难的问题上，专家人设在具有有效答案的条目上达到满分精度。
• 通过对专家人设的置信度增强消除了基线错误。
• 法证分析显示最难的 GPQA 项目中有一半包含化学或逻辑上不可辩护的答案，影响评估结果。
• 模型的 CoT 显示推理偏离不可能的答案，导致对准确的化学知识进行惩罚。