[Paper Review] Human Following for Wheeled Robot with Monocular Pan-tilt Camera

Zheng Zhu, Hongxuan Ma+1 more|arXiv (Cornell University)|Sep 13, 2019

Video Surveillance and Tracking Methods61 references4 citations

TL;DR

This paper presents a monocular pan-tilt camera-based human following system for wheeled robots that eliminates the need for depth sensors by leveraging human height constraints and a unified controller. The system combines Siamese networks with optical flow for robust visual tracking (FlowTrack++), enabling real-time, accurate target localization and motion control, achieving stable following in both simulation and real-world experiments with minimal tracking error.

ABSTRACT

Human following on mobile robots has witnessed significant advances due to its potentials for real-world applications. Currently most human following systems are equipped with depth sensors to obtain distance information between human and robot, which suffer from the perception requirements and noises. In this paper, we design a wheeled mobile robot system with monocular pan-tilt camera to follow human, which can stay the target in the field of view and keep following simultaneously. The system consists of fast human detector, real-time and accurate visual tracker, and unified controller for mobile robot and pan-tilt camera. In visual tracking algorithm, both Siamese networks and optical flow information are exploited to locate and regress human simultaneously. In order in perform following with a monocular camera, the constraint of human height is introduced to design the controller. In experiments, human following are conducted and analysed in simulations and a real robot platform, which demonstrate the effectiveness and robustness of the overall system.

Motivation & Objective

• To develop a human following system for wheeled robots that does not rely on depth sensors, reducing hardware complexity and noise susceptibility.
• To maintain the human target within the camera’s field of view while keeping the target’s image height close to a predefined constant, ensuring consistent tracking.
• To design a unified controller for both robot motion and pan-tilt camera movement using monocular vision and human height constraints.
• To improve tracking robustness under occlusions, scale variations, and background changes using a novel visual tracker (FlowTrack++).

Proposed method

• A fast YOLO-based detector initializes the human bounding box in the first frame.
• The FlowTrack++ visual tracker combines Siamese networks and optical flow to simultaneously locate and regress the human target in real time.
• Hard-negative mining is applied to suppress responses to distractors and improve tracking stability.
• A failure recovery strategy is implemented to handle occlusions and out-of-view events.
• A unified controller uses image-based error signals (e_u, e_v, e_v2) and height constraint (h ≈ H) to generate robot velocity (V_r) and angular velocity (ω_r), along with pan/tilt angles (α, β).
• The controller leverages the known human height (1.8m) to estimate depth from monocular image height, enabling distance-informed control without depth sensors.

Experimental results

Research questions

• RQ1 Can a monocular camera system effectively track a human target in dynamic environments without depth sensors?
• RQ2 How can a unified controller for robot motion and pan-tilt camera be designed using only monocular image feedback and human height constraints?
• RQ3 To what extent does combining Siamese networks and optical flow improve tracking robustness under occlusion and scale variation?
• RQ4 How does the proposed FlowTrack++ tracker compare to ECO and GOTURN in real-world human following scenarios?

Key findings

• In simulations, the controller successfully maintained the target near the image center with stable height (h ≈ 100 pixels) and smooth robot trajectories, even when the target moved in a circular path.
• In indoor experiments, FlowTrack++ kept the target height h near 500 pixels with e_u and e_v consistently near zero, while ECO and GOTURN showed significant fluctuations.
• In outdoor experiments, FlowTrack++ maintained e_u and e_v within ±20 pixels and h around 300 pixels, demonstrating robustness under real-world challenges like lighting changes and background clutter.
• FlowTrack++ effectively handled pose changes, scale variations, occlusions, and distractors due to its flow aggregation and hard-negative mining mechanisms.
• The failure recovery strategy enabled quick reacquisition of the target after occlusion, reducing tracking drift and improving system reliability.