[Paper Review] The Matter of Time -- A General and Efficient System for Precise Sensor Synchronization in Robotic Computing

Shaoshan Liu, Yu Bo+6 more|arXiv (Cornell University)|Mar 30, 2021

Robotics and Automated Systems10 references4 citations

TL;DR

This paper presents a general, FPGA-based hardware synchronization system for robotic computing that achieves microsecond-level precision by timestamping sensor data at the SoC interface, minimizing software-induced latency. The system ensures high accuracy (±100 μs for LiDAR, cycle-level for camera/IMU) with low resource consumption (6.9K LUTs, 7.1KRegs), enabling reliable intra- and inter-machine synchronization in autonomous vehicles and robots.

ABSTRACT

Time synchronization is a critical task in robotic computing such as autonomous driving. In the past few years, as we developed advanced robotic applications, our synchronization system has evolved as well. In this paper, we first introduce the time synchronization problem and explain the challenges of time synchronization, especially in robotic workloads. Summarizing these challenges, we then present a general hardware synchronization system for robotic computing, which delivers high synchronization accuracy while maintaining low energy and resource consumption. The proposed hardware synchronization system is a key building block in our future robotic products.

Motivation & Objective

• Address the critical challenge of precise time synchronization in robotic systems, where sensor misalignment leads to inaccurate perception and localization.
• Overcome the limitations of software-based timestamping, which introduces variable latency and reduces synchronization accuracy.
• Design a general-purpose hardware solution that supports both intra-machine (within a single robot) and inter-machine (across multiple robots) synchronization.
• Ensure low energy and resource consumption to enable deployment in embedded robotic systems.
• Extend the system's applicability to space exploration by adapting it to use SCLK instead of GPS as a global time reference.

Proposed method

• Implement a synchronization system on a Xilinx Zynq FPGA-based SoC, leveraging both ARM CPU clusters and FPGA fabric for hybrid processing.
• Obtain timestamps at the sensor interface (e.g., MIPI, CAN, Ethernet, serial ports) rather than at the host application, reducing timing jitter and latency.
• Use hardware-triggering circuits to generate synchronized pulses for cameras and IMUs, achieving cycle-level precision in timestamping.
• Integrate the Precision Time Protocol (PTP) for LiDAR frame synchronization, achieving sub-100 μs variation across frames.
• Customize FPGA logic for GPS, IMU, CAN, and triggering circuits, while reusing Xilinx IP cores for MIPI and Ethernet interfaces.
• Deploy the Arm cores for protocol handling, configuration management, and user application interfaces, decoupling control from real-time synchronization logic.

Experimental results

Research questions

• RQ1 How can sensor synchronization be made both highly accurate and efficient in real-world robotic systems with diverse sensor types?
• RQ2 What are the key design principles that enable a general-purpose synchronization system for both intra-machine and inter-machine scenarios?
• RQ3 To what extent can hardware timestamping at the SoC interface reduce timing jitter compared to software-based approaches?
• RQ4 Can the system maintain high accuracy across different sensors (camera, IMU, LiDAR, Radar) with minimal resource overhead?
• RQ5 How can the system be adapted for use in space environments where GPS is unavailable and SCLK is used as the time reference?

Key findings

• The system achieves cycle-level precision for camera and IMU timestamping by triggering sensors and capturing timestamps at the SoC interface.
• LiDAR frame synchronization via PTP exhibits timing variations within 100 μs, ensuring high consistency across frames.
• The hardware implementation consumes only 6.9K LUTs and 7.1K registers, demonstrating low resource overhead for embedded deployment.
• A 40 ms mis-synchronization between camera and IMU leads to a 10 m translational error and up to 3° rotational error in VIO pose estimation, highlighting the critical need for precise synchronization.
• The system supports both intra-machine and inter-machine synchronization, with a design that can be adapted to space missions by replacing GPS with SCLK as the global time source.
• Tolerable synchronization errors for object detection are as low as 34 ms at 40 m/s when requiring 0.5 IoU, underscoring the importance of sub-millisecond accuracy in high-speed applications.