[Paper Review] T-Rex: Counting by Visual Prompting

Qing Jiang, Feng Li+5 more|arXiv (Cornell University)|Nov 22, 2023

Video Surveillance and Tracking Methods5 citations

TL;DR

T-Rex is a novel interactive object counting model that leverages visual prompting to detect and count objects in images with high accuracy and interpretability. By allowing users to specify target objects via points or bounding boxes on a reference image, T-Rex detects similar instances in the target image and aggregates them for counting, achieving state-of-the-art zero-shot performance on diverse benchmarks including the new CA-44 dataset.

ABSTRACT

We introduce T-Rex, an interactive object counting model designed to first detect and then count any objects. We formulate object counting as an open-set object detection task with the integration of visual prompts. Users can specify the objects of interest by marking points or boxes on a reference image, and T-Rex then detects all objects with a similar pattern. Guided by the visual feedback from T-Rex, users can also interactively refine the counting results by prompting on missing or falsely-detected objects. T-Rex has achieved state-of-the-art performance on several class-agnostic counting benchmarks. To further exploit its potential, we established a new counting benchmark encompassing diverse scenarios and challenges. Both quantitative and qualitative results show that T-Rex possesses exceptional zero-shot counting capabilities. We also present various practical application scenarios for T-Rex, illustrating its potential in the realm of visual prompting.

Motivation & Objective

• Address the limitations of existing object counting methods, which are often constrained by fixed categories, lack interpretability, or rely on imperfect textual descriptions.
• Overcome the challenges of zero-shot counting for objects without concise textual descriptions by introducing visual prompting as a more effective specification method.
• Develop an interactive, open-set counting system that allows users to refine results through visual feedback, improving accuracy and user confidence.
• Establish a new, comprehensive benchmark (CA-44) to evaluate zero-shot counting performance across diverse domains and complex scenarios.
• Demonstrate the practical utility of T-Rex in real-world applications such as agriculture, medicine, logistics, and retail through extensive case studies.

Proposed method

• Formulate object counting as an open-set object detection task enhanced with visual prompting, enabling detection of any object without predefined category constraints.
• Use user-provided visual prompts (points or bounding boxes) on a reference image to guide the model in detecting similar patterns in the target image.
• Leverage a detection-based architecture that processes visual prompts through a vision encoder and a lightweight decoder for fast inference and interactive refinement.
• Integrate visual feedback from detection results to allow users to iteratively correct missed or false detections by adding new prompts.
• Utilize segmentation models like SAM to generate masks on detected boxes for improved visualization and user verification.
• Train and evaluate on both existing class-agnostic counting benchmarks and a new, large-scale benchmark (CA-44) with 44 diverse datasets across eight domains.

Experimental results

Research questions

• RQ1 Can visual prompting significantly improve zero-shot object counting performance compared to text-based prompting in open-vocabulary detection?
• RQ2 How effective is an interactive, user-guided refinement process in correcting detection errors and improving counting accuracy?
• RQ3 To what extent can a detection-based, visual prompting model generalize across diverse, real-world counting scenarios without fine-tuning?
• RQ4 How does T-Rex compare to state-of-the-art multi-modal LLMs like GPT-4V in terms of counting accuracy and reliability?
• RQ5 What are the key failure modes of visual prompting in complex scenes, and how can they be mitigated?

Key findings

• T-Rex achieves state-of-the-art performance on two existing class-agnostic counting benchmarks, outperforming both density map regression and closed-set detection methods.
• On the newly introduced CA-44 benchmark, T-Rex significantly outperforms baseline models including Grounding DINO and GPT-4V across all 44 datasets and eight domains.
• T-Rex demonstrates superior zero-shot counting accuracy compared to GPT-4V, especially in one-shot visual prompting settings, indicating that visual prompts are more effective than text prompts for precise object perception.
• The interactive prompting mechanism enables users to correct detection errors efficiently, with minimal computational cost per iteration, maintaining high speed and resource efficiency.
• Failure cases reveal limitations in single-target scenes and dense multi-object scenes, where overfitting and false positives occur, highlighting the need for multi-prompt or negative prompting strategies.
• T-Rex shows strong practical potential across diverse domains, including agriculture, medicine, logistics, and retail, as validated through real-world application case studies.