[Paper Review] Visual Agents as Fast and Slow Thinkers

Guangyan Sun, Mingyu Jin+8 more|arXiv (Cornell University)|Aug 16, 2024

Reinforcement Learning in Robotics4 citations

TL;DR

FaST introduces a visual agent framework that dynamically switches between fast (System 1) and slow (System 2) thinking modes using a switch adapter to improve reasoning in visual tasks. By integrating proposal and segmentation adapters, it enhances accuracy and interpretability, achieving 80.8% VQA accuracy and 48.7% GIoU on ReasonSeg, outperforming strong baselines with improved robustness and reduced hallucination.

ABSTRACT

Achieving human-level intelligence requires refining cognitive distinctions between System 1 and System 2 thinking. While contemporary AI, driven by large language models, demonstrates human-like traits, it falls short of genuine cognition. Transitioning from structured benchmarks to real-world scenarios presents challenges for visual agents, often leading to inaccurate and overly confident responses. To address the challenge, we introduce FaST, which incorporates the Fast and Slow Thinking mechanism into visual agents. FaST employs a switch adapter to dynamically select between System 1/2 modes, tailoring the problem-solving approach to different task complexity. It tackles uncertain and unseen objects by adjusting model confidence and integrating new contextual data. With this novel design, we advocate a flexible system, hierarchical reasoning capabilities, and a transparent decision-making pipeline, all of which contribute to its ability to emulate human-like cognitive processes in visual intelligence. Empirical results demonstrate that FaST outperforms various well-known baselines, achieving 80.8% accuracy over VQA^{v2} for visual question answering and 48.7% GIoU score over ReasonSeg for reasoning segmentation, demonstrate FaST's superior performance. Extensive testing validates the efficacy and robustness of FaST's core components, showcasing its potential to advance the development of cognitive visual agents in AI systems. The code is available at ttps://github.com/GuangyanS/Sys2-LLaVA.

Motivation & Objective

• To address the limitations of current visual agents in real-world environments, where models often produce overconfident, hallucinated responses.
• To bridge the gap between artificial intelligence and human-level cognition by explicitly modeling System 1 (fast, intuitive) and System 2 (slow, deliberate) thinking.
• To improve reasoning robustness in visual tasks involving uncertainty, invisibility, or complex queries.
• To develop a transparent, hierarchical reasoning pipeline with interpretable intermediate outputs for enhanced explainability.

Proposed method

• A switch adapter dynamically selects between System 1 and System 2 modes based on task complexity and model confidence.
• System 1 is triggered for simple tasks with high confidence, enabling fast inference.
• System 2 is activated when uncertainty or invisibility is detected—e.g., ambiguous queries or small objects—using negative contextual data to recalibrate confidence.
• A proposal adapter generates region-of-interest (RoI) candidates to guide detailed analysis.
• A segmentation adapter produces pixel-level masks for fine-grained reasoning on complex instances.
• The framework integrates world knowledge and contextual clues to build a chain of evidence, enhancing interpretability and reducing hallucinations.

Experimental results

Research questions

• RQ1 How can visual agents dynamically adapt their reasoning mode based on task complexity and uncertainty?
• RQ2 To what extent can a hybrid fast/slow thinking mechanism reduce hallucination in visual question answering and segmentation?
• RQ3 What role do contextual clues and pixel-level adapters play in improving reasoning robustness for unseen or ambiguous visual inputs?
• RQ4 How does the integration of a transparent, neuro-symbolic reasoning pipeline enhance model explainability and performance?

Key findings

• FaST achieves 80.8% accuracy on VQA-v2, significantly outperforming baseline models.
• On the ReasonSeg benchmark, FaST attains a 48.7% GIoU score, demonstrating superior instance-level reasoning and segmentation accuracy.
• The ablation study confirms that both the proposal adapter and segmentation adapter contribute meaningfully to performance gains, with combined use yielding the best results.
• Incorporating missing objects and context clues into the switch adapter improves performance on MME and refCOCOg by up to 5.6 points.
• FaST operates approximately 31% faster than pure System 2 inference on MME and 50% faster on GQA, maintaining high performance.
• Qualitative results show FaST produces more accurate, localized, and explainable predictions than LLaVA-v1.5 and LISA-7B, especially in complex or ambiguous scenarios.