[Paper Review] Follow Anything: Open-set detection, tracking, and following in real-time

Alaa Maalouf, Ninad Jadhav+6 more|arXiv (Cornell University)|Aug 10, 2023

Anomaly Detection Techniques and ApplicationsComputer Science3 citations

TL;DR

This paper introduces FAn, a real-time, open-vocabulary, multimodal robotic system that detects, tracks, and follows any object using text, image, or click queries. Leveraging DINO and SAM for visual feature extraction and segmentation, it achieves 6–20 FPS on a laptop with a 6–8 GB GPU, enabling robust tracking even after occlusion through re-detection mechanisms.

ABSTRACT

Tracking and following objects of interest is critical to several robotics use cases, ranging from industrial automation to logistics and warehousing, to healthcare and security. In this paper, we present a robotic system to detect, track, and follow any object in real-time. Our approach, dubbed ``follow anything'' (FAn), is an open-vocabulary and multimodal model -- it is not restricted to concepts seen at training time and can be applied to novel classes at inference time using text, images, or click queries. Leveraging rich visual descriptors from large-scale pre-trained models (foundation models), FAn can detect and segment objects by matching multimodal queries (text, images, clicks) against an input image sequence. These detected and segmented objects are tracked across image frames, all while accounting for occlusion and object re-emergence. We demonstrate FAn on a real-world robotic system (a micro aerial vehicle) and report its ability to seamlessly follow the objects of interest in a real-time control loop. FAn can be deployed on a laptop with a lightweight (6-8 GB) graphics card, achieving a throughput of 6-20 frames per second. To enable rapid adoption, deployment, and extensibility, we open-source all our code on our project webpage at https://github.com/alaamaalouf/FollowAnything . We also encourage the reader to watch our 5-minutes explainer video in this https://www.youtube.com/watch?v=6Mgt3EPytrw .

Motivation & Objective

• To address the limitations of closed-set robotic object-following systems that require prior training on fixed object categories.
• To enable flexible, multimodal user interaction—via text, images, or clicks—without requiring retraining for new object classes.
• To develop a real-time, lightweight system deployable on resource-constrained robotic platforms such as micro aerial vehicles.
• To ensure robustness during occlusion and tracking loss through autonomous re-detection mechanisms.
• To bridge state-of-the-art vision foundation models with real-time robotic control loops for practical deployment.

Proposed method

• FAn uses a multimodal query interface allowing users to specify the object of interest via text, image, bounding box, or click.
• It employs Segment Anything Model (SAM) to generate multiple candidate masks from the input image, followed by DINO-based visual feature extraction for pixel-wise embedding.
• Object detection is performed by computing cosine similarity between DINO patch embeddings and the query embedding (from text or image), selecting the mask with the closest match.
• The system integrates a real-time tracking pipeline that maintains object state across frames, with re-detection mechanisms to recover from occlusion or tracking failure.
• A unified control loop enables visual servoing, ensuring the object remains within the camera’s field of view on a micro aerial vehicle.
• The system is optimized for inference efficiency, achieving 6–20 FPS on a laptop with a 6–8 GB GPU, and is open-sourced for extensibility.

Experimental results

Research questions

• RQ1 Can a robotic system detect and follow any object in real-time without requiring prior training on specific categories?
• RQ2 How effective is multimodal querying (text, image, click) in enabling open-set object following with high accuracy and low latency?
• RQ3 What is the performance trade-off between using DINO-SOLO and SAM+CLIP for visual feature extraction and detection in low-resource settings?
• RQ4 How well does the system handle object occlusion and tracking recovery in dynamic, real-world environments?
• RQ5 To what extent can a lightweight, open-source system be deployed on edge devices like laptops or micro aerial vehicles for real-time robotic control?

Key findings

• FAn achieves real-time performance of 6–20 FPS on a laptop with a 6–8 GB GPU, enabling deployment on resource-constrained robotic platforms.
• The system successfully detects and tracks objects specified via text, image, or click queries, even for novel categories not seen during training.
• Using DINO for visual feature extraction outperforms CLIP-based methods in speed and feature quality, particularly on small or low-resolution object crops.
• DINO-SOLO provides more robust detection than SAM+CLIP in low-resolution or ambiguous scenes, though with lower mask quality.
• The re-detection mechanism effectively restores tracking after occlusion, maintaining system continuity and reliability.
• The system demonstrates strong generalization across diverse real-world scenarios, including following drones, RC cars, and manually moved bricks, with minimal user input.