[Paper Review] Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra

Darioush Kevian, Usman Syed+6 more|arXiv (Cornell University)|Apr 4, 2024

Reservoir Engineering and Simulation Methods18 citations

TL;DR

The paper benchmarks GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra on ControlBench, a dataset of undergraduate control problems, showing Claude 3 Opus generally outperforms the others, with notable challenges in visual data interpretation.

ABSTRACT

In this paper, we explore the capabilities of state-of-the-art large language models (LLMs) such as GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra in solving undergraduate-level control problems. Controls provides an interesting case study for LLM reasoning due to its combination of mathematical theory and engineering design. We introduce ControlBench, a benchmark dataset tailored to reflect the breadth, depth, and complexity of classical control design. We use this dataset to study and evaluate the problem-solving abilities of these LLMs in the context of control engineering. We present evaluations conducted by a panel of human experts, providing insights into the accuracy, reasoning, and explanatory prowess of LLMs in control engineering. Our analysis reveals the strengths and limitations of each LLM in the context of classical control, and our results imply that Claude 3 Opus has become the state-of-the-art LLM for solving undergraduate control problems. Our study serves as an initial step towards the broader goal of employing artificial general intelligence in control engineering.

Motivation & Objective

• Introduce ControlBench, a natural-language control problem dataset reflecting undergraduate control design breadth and complexity.
• Evaluate leading LLMs (GPT-4, Claude 3 Opus, Gemini 1.0 Ultra) on ControlBench via human expert assessment.
• Analyze accuracy, reasoning quality, and explanations, plus model-specific strengths and limitations.
• Explore self-correction and the impact of visual data (plots) on model performance.
• Provide a simplified ControlBench-C for rapid, non-expert evaluation.

Proposed method

• Construct ControlBench with 147 undergraduate control problems covering topics like stability, time response, Bode/Nyquist plots, loop-shaping, and advanced topics.
• Annotate problems in LaTeX with detailed step-by-step solutions for reproducibility.
• Evaluate three LLMs in zero-shot and self-correcting settings via human expert scoring of accuracy (ACC) and self-corrected accuracy (ACC-s).
• Analyze error modes and visual data misreading to identify bottlenecks and improvement avenues.
• Present a reduced ControlBench-C variant for quick automatic evaluation.

Experimental results

Research questions

• RQ1 How do GPT-4, Claude 3 Opus, and Gemini 1.0 Ultra perform on undergraduate control problems in ControlBench?
• RQ2 Which model shows the strongest accuracy and self-correction capabilities across control topics?
• RQ3 What are the main failure modes when LLMs solve control problems, and how does visual data interpretation affect performance?
• RQ4 Can a simplified multiple-choice version (ControlBench-C) reliably benchmark LLMs without control background?
• RQ5 What insights do the results provide for integrating LLMs into control engineering education and workflows?

Key findings

• Claude 3 Opus achieves the highest overall ACC and ACC-s across topics, indicating superior accuracy and self-correction.
• GPT-4 and Claude 3 Opus perform well on background mathematics, stability, and time-response problems, with Claude 3 Opus generally leading on visual-component tasks.
• Gemini 1.0 Ultra lags behind the others in overall performance and excels less consistently across topics.
• All models struggle with reading graphical data such as Bode, Nyquist, and root-locus plots, highlighting limitations in visual language understanding.
• Self-correction prompts significantly improve ACC-s across models, demonstrating practical value in iterative reasoning.
• ControlBench-C offers a faster, but narrower, evaluation of LLM capabilities and may not capture comprehensive reasoning.