[Paper Review] Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases

Zhangyang Qi, Fang Ye+7 more|arXiv (Cornell University)|Dec 22, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper conducts a comprehensive qualitative comparison of Google's Gemini and OpenAI's GPT-4V across vision-language understanding, reasoning, and multimodal tasks. It demonstrates that GPT-4V excels in precision and object recognition, while Gemini provides richer, link-enhanced responses; combining both models leverages their complementary strengths, significantly improving performance in product recommendation and multi-image story generation.

ABSTRACT

The rapidly evolving sector of Multi-modal Large Language Models (MLLMs) is at the forefront of integrating linguistic and visual processing in artificial intelligence. This paper presents an in-depth comparative study of two pioneering models: Google's Gemini and OpenAI's GPT-4V(ision). Our study involves a multi-faceted evaluation of both models across key dimensions such as Vision-Language Capability, Interaction with Humans, Temporal Understanding, and assessments in both Intelligence and Emotional Quotients. The core of our analysis delves into the distinct visual comprehension abilities of each model. We conducted a series of structured experiments to evaluate their performance in various industrial application scenarios, offering a comprehensive perspective on their practical utility. We not only involve direct performance comparisons but also include adjustments in prompts and scenarios to ensure a balanced and fair analysis. Our findings illuminate the unique strengths and niches of both models. GPT-4V distinguishes itself with its precision and succinctness in responses, while Gemini excels in providing detailed, expansive answers accompanied by relevant imagery and links. These understandings not only shed light on the comparative merits of Gemini and GPT-4V but also underscore the evolving landscape of multimodal foundation models, paving the way for future advancements in this area. After the comparison, we attempted to achieve better results by combining the two models. Finally, We would like to express our profound gratitude to the teams behind GPT-4V and Gemini for their pioneering contributions to the field. Our acknowledgments are also extended to the comprehensive qualitative analysis presented in 'Dawn' by Yang et al. This work, with its extensive collection of image samples, prompts, and GPT-4V-related results, provided a foundational basis for our analysis.

Motivation & Objective

• To conduct a detailed, qualitative comparison of Gemini and GPT-4V across diverse vision-language understanding and reasoning benchmarks.
• To evaluate the practical utility of both models in real-world industrial applications such as defect detection, grocery checkout, and GUI navigation.
• To investigate the synergistic potential of combining GPT-4V and Gemini to overcome individual limitations and enhance multimodal performance.
• To analyze differences in emotional intelligence, temporal understanding, and multilingual capabilities between the two models.
• To explore novel integration paradigms that leverage the distinct strengths of each model in complex, multi-stage tasks.

Proposed method

• Conducted structured, multi-faceted evaluations across 12 core categories, including image recognition, text understanding in images, reasoning, and temporal video understanding.
• Employed controlled prompt variations and scenario adjustments to ensure fair and balanced comparisons between Gemini and GPT-4V.
• Implemented a two-stage model fusion strategy: first using GPT-4V for accurate object detection and description, then feeding those outputs into Gemini for link-based retrieval and narrative generation.
• Evaluated model performance in industrial scenarios such as embodied agents, GUI navigation, and document reasoning using real-world image and text inputs.
• Used qualitative case studies—such as product recommendation and multi-image story generation—to demonstrate the benefits of model combination.
• Leveraged the 'Dawn' dataset and analysis framework from Yang et al. (2023) as a foundational reference for prompt design and evaluation consistency.

Experimental results

Research questions

• RQ1 How do Gemini and GPT-4V compare in basic object recognition, landmark detection, and food identification tasks?
• RQ2 In what ways do the two models differ in understanding complex visual elements such as equations, charts, and abstract images?
• RQ3 How do Gemini and GPT-4V perform in multimodal reasoning tasks, including emotional intelligence tests and detective-style reasoning?
• RQ4 Can the integration of GPT-4V and Gemini improve performance beyond individual model capabilities in real-world applications?
• RQ5 What are the strengths and limitations of each model in industrial applications like auto insurance claims processing and GUI navigation?

Key findings

• GPT-4V demonstrated superior precision and succinctness in object recognition and text understanding, particularly in complex scenarios like equation and table recognition.
• Gemini outperformed GPT-4V in generating detailed, expansive responses and providing relevant web links for product recommendations.
• In integrated tasks such as product identification, combining GPT-4V’s accurate description with Gemini’s retrieval capability significantly improved link recommendation accuracy.
• For multi-image story generation, GPT-4V accurately summarized sub-images, while Gemini generated coherent, stylistically aligned narratives, outperforming single-model approaches.
• Gemini’s inability to process multiple images simultaneously limited its performance in tasks requiring holistic scene understanding compared to GPT-4V.
• In industrial applications like GUI navigation and embodied agents, GPT-4V consistently outperformed Gemini due to better spatial and sequential reasoning capabilities.