[Paper Review] Task-relevant Representation Learning for Networked Robotic Perception

Manabu Nakanoya, Sandeep Chinchali+4 more|arXiv (Cornell University)|Nov 6, 2020

Robotics and Automated Systems22 references4 citations

TL;DR

This paper proposes a co-design framework that learns task-relevant, minimal representations of robotic sensory data by aligning encoder/decoder networks with a pre-trained, fixed task network. By using the task objective to guide compression, the method achieves up to 11× higher compression than baseline autoencoders while maintaining high accuracy on diverse tasks like Mars terrain classification, neural motion planning, and environmental anomaly detection, even with low-power edge accelerators.

ABSTRACT

Today, even the most compute-and-power constrained robots can measure complex, high data-rate video and LIDAR sensory streams. Often, such robots, ranging from low-power drones to space and subterranean rovers, need to transmit high-bitrate sensory data to a remote compute server if they are uncertain or cannot scalably run complex perception or mapping tasks locally. However, today's representations for sensory data are mostly designed for human, not robotic, perception and thus often waste precious compute or wireless network resources to transmit unimportant parts of a scene that are unnecessary for a high-level robotic task. This paper presents an algorithm to learn task-relevant representations of sensory data that are co-designed with a pre-trained robotic perception model's ultimate objective. Our algorithm aggressively compresses robotic sensory data by up to 11x more than competing methods. Further, it achieves high accuracy and robust generalization on diverse tasks including Mars terrain classification with low-power deep learning accelerators, neural motion planning, and environmental timeseries classification.

Motivation & Objective

• To address the inefficiency of current sensory data representations, which are optimized for human perception rather than robotic task performance.
• To reduce communication and computational costs for resource-constrained robots transmitting data to remote servers for perception tasks.
• To co-design minimal, task-specific sensory representations that preserve only salient features relevant to a pre-trained task model’s objective.
• To enable efficient deployment on low-power edge devices like the Google Edge TPU by minimizing on-board computation and bandwidth use.
• To generalize across multiple sensor modalities, including RGB-D video, LIDAR point clouds, and environmental timeseries.

Proposed method

• A differentiable, pre-trained task network is fixed at a central server, and its objective guides the learning of a robot-side encoder and server-side decoder.
• The encoder compresses raw sensory input into a low-dimensional bottleneck representation $ z $, while the decoder reconstructs the input for task inference.
• The loss function combines task-specific classification loss and reconstruction loss, with the task loss dominating to prioritize information relevant to the downstream task.
• The framework supports off-the-shelf task modules without requiring input dimension changes, enabling plug-and-play integration with public models.
• The method is trained end-to-end using backpropagation, with the task network parameters frozen during training.
• Computation is flexibly allocated between robot and server, enabling compatibility with low-power DNN accelerators.

Experimental results

Research questions

• RQ1 Can a co-designed encoder-decoder framework learn minimal, task-relevant representations that outperform standard autoencoders in compression efficiency?
• RQ2 How well does the proposed method generalize to out-of-domain robotic perception tasks, such as unseen Mars terrain or environmental sensor anomalies?
• RQ3 To what extent can the method reduce communication and on-board computation costs while maintaining high task accuracy on low-power hardware?
• RQ4 Can the framework be applied across diverse sensor modalities, including video, LIDAR, and timeseries data, without architectural changes?
• RQ5 How does task-guided compression compare to reconstruction-focused compression in terms of robustness and efficiency?

Key findings

• The method achieves up to 11× higher compression than standard autoencoders by focusing on task-relevant features rather than full reconstruction.
• On Mars terrain classification, TaskNet outperforms MPNet by a significant margin at small bottleneck sizes ($ z = 4, 8 $), demonstrating superior generalization on out-of-domain terrain.
• For neural motion planning, TaskNet maintains high performance even with $ z = 8 $, while MPNet degrades sharply, indicating better information efficiency.
• In environmental timeseries anomaly detection, TaskNet matches the 90% test accuracy of the original data while achieving low reconstruction loss and high compression.
• The framework enables deployment on low-power edge devices like the Google Edge TPU, making it suitable for real-world robotic systems with strict compute and bandwidth constraints.
• The approach generalizes across multiple sensor modalities, including video, LIDAR, and multi-sensor timeseries, with consistent performance gains.