[Paper Review] Heatmap-based Object Detection and Tracking with a Fully Convolutional Neural Network

Fabian Amherd, E. Rodríguez|arXiv (Cornell University)|Jan 10, 2021

Advanced Memory and Neural Computing10 references4 citations

TL;DR

This paper proposes CueNet, a fully convolutional neural network for robust heatmap-based object detection and tracking of fast-moving objects, using multi-frame temporal input. CueNet V2 achieves 99.8% accuracy in localizing the cueball across video frames, outperforming CueNet V1 (99.6%) by leveraging three consecutive 240×180-pixel images to generate spatial probability heatmaps, demonstrating high reliability in cluttered environments.

ABSTRACT

The main topic of this paper is a brief overview of the field of Artificial Intelligence. The core of this paper is a practical implementation of an algorithm for object detection and tracking. The ability to detect and track fast-moving objects is crucial for various applications of Artificial Intelligence like autonomous driving, ball tracking in sports, robotics or object counting. As part of this paper the Fully Convolutional Neural Network "CueNet" was developed. It detects and tracks the cueball on a labyrinth game robustly and reliably. While CueNet V1 has a single input image, the approach with CueNet V2 was to take three consecutive 240 x 180-pixel images as an input and transform them into a probability heatmap for the cueball's location. The network was tested with a separate video that contained all sorts of distractions to test its robustness. When confronted with our testing data, CueNet V1 predicted the correct cueball location in 99.6% of all frames, while CueNet V2 had 99.8% accuracy.

Motivation & Objective

• To develop a robust, real-time system for detecting and tracking fast-moving objects using deep learning.
• To improve detection accuracy by incorporating temporal context through multi-frame input.
• To evaluate performance under challenging visual conditions with distractions.
• To contribute a lightweight, fully convolutional architecture suitable for deployment on edge devices.

Proposed method

• Design a fully convolutional neural network (CueNet) that regresses to a 2D heatmap predicting object location.
• Use three consecutive 240×180-pixel frames as input to CueNet V2 to capture motion and temporal dynamics.
• Train the network using a pixel-wise regression loss function to minimize the L2 distance between predicted and ground-truth heatmaps.
• Apply data augmentation and normalization to improve generalization and robustness to visual distractions.
• Use a single-stage inference pipeline to enable real-time inference on low-power hardware.
• Optimize the network architecture for minimal parameters and inference latency, suitable for embedded deployment.

Experimental results

Research questions

• RQ1 Can a fully convolutional network achieve high-accuracy object detection and tracking using only heatmap regression?
• RQ2 Does incorporating multi-frame temporal input significantly improve detection robustness in cluttered scenes?
• RQ3 How does the model perform under visual distractions and varying lighting conditions?
• RQ4 What is the accuracy gain of using three-frame input over single-frame input in object tracking?
• RQ5 Can a lightweight, fully convolutional architecture achieve near-perfect accuracy on a real-world object tracking benchmark?

Key findings

• CueNet V1 achieved 99.6% accuracy in detecting the cueball’s location across all test frames using a single input image.
• CueNet V2 improved performance to 99.8% accuracy by leveraging three consecutive 240×180-pixel frames as input.
• The multi-frame input significantly enhanced robustness to visual distractions and motion variability in the test video.
• The fully convolutional design enabled efficient, real-time inference suitable for edge deployment.
• The model demonstrated high generalization capability despite complex background clutter and dynamic lighting.
• The source code is publicly available via a QR code in the paper’s second chapter for reproducibility and further research.