[Paper Review] Live Target Detection with Deep Learning Neural Network and Unmanned Aerial Vehicle on Android Mobile Device

Ali Canberk Anar, Erkan Bostancı+1 more|arXiv (Cornell University)|Mar 19, 2018

Robotics and Sensor-Based Localization2 references3 citations

TL;DR

This paper presents an Android-based live target detection system using a DJI Phantom 3 Professional drone and a TensorFlow-powered deep learning model running on mobile hardware. It achieves real-time inference (851ms) with high confidence classifications, such as 31.75% proximity for 'sports car' in a parking lot scene, demonstrating feasible on-device UAV vision processing for mobile platforms.

ABSTRACT

This paper describes the stages faced during the development of an Android program which obtains and decodes live images from DJI Phantom 3 Professional Drone and implements certain features of the TensorFlow Android Camera Demo application. Test runs were made and outputs of the application were noted. A lake was classified as seashore, breakwater and pier with the proximities of 24.44%, 21.16% and 12.96% respectfully. The joystick of the UAV controller and laptop keyboard was classified with the proximities of 19.10% and 13.96% respectfully. The laptop monitor was classified as screen, monitor and television with the proximities of 18.77%, 14.76% and 14.00% respectfully. The computer used during the development of this study was classified as notebook and laptop with the proximities of 20.04% and 11.68% respectfully. A tractor parked at a parking lot was classified with the proximity of 12.88%. A group of cars in the same parking lot were classified as sports car, racer and convertible with the proximities of 31.75%, 18.64% and 13.45% respectfully at an inference time of 851ms.

Motivation & Objective

• To develop a real-time, on-device target detection system for live video streams from a UAV on an Android mobile platform.
• To integrate TensorFlow's mobile inference engine with live video feed from a DJI Phantom 3 Professional drone.
• To evaluate the performance of on-device deep learning inference in classifying real-world objects under varying environmental conditions.
• To demonstrate the feasibility of deploying lightweight CNN models on mobile devices for UAV-based visual perception.

Proposed method

• The system captures live video from a DJI Phantom 3 Professional drone via USB connection to an Android device.
• Raw video frames are decoded and preprocessed for input into a pre-trained TensorFlow MobileNet SSD model.
• The inference engine runs on the Android device using the TensorFlow Lite framework for on-device processing.
• Class predictions are generated with top-5 class probabilities, reported as proximity scores to the true label.
• The system leverages the TensorFlow Android Camera Demo as a base framework for camera integration and real-time processing.
• Results are logged and analyzed for various real-world targets, including vehicles, electronic devices, and natural structures.

Experimental results

Research questions

• RQ1 Can a lightweight deep learning model achieve real-time object detection on an Android mobile device using live video from a commercial UAV?
• RQ2 How accurate are on-device inference results when classifying diverse real-world targets such as vehicles, electronic devices, and structures?
• RQ3 What is the inference latency of a mobile-optimized SSD model when processing live UAV video on consumer-grade Android hardware?
• RQ4 How do proximity scores (confidence levels) vary across different object categories under real-world conditions?
• RQ5 To what extent can a mobile platform replace desktop-based processing for UAV-based visual detection tasks?

Key findings

• The system achieved real-time inference with an average latency of 851ms for object detection on a mobile Android device.
• A group of cars in a parking lot were classified as 'sports car' with a proximity score of 31.75%, indicating strong model confidence.
• A tractor was classified with a proximity of 12.88%, reflecting moderate confidence in detection.
• The laptop monitor was classified as 'screen', 'monitor', and 'television' with proximities of 18.77%, 14.76%, and 14.00% respectively.
• The joystick of the UAV controller was classified with 19.10% proximity, showing reasonable detection performance for small, non-standard objects.
• The computer used in development was classified as 'notebook' with 20.04% proximity, indicating effective recognition of portable computing devices.