[Paper Review] Accelerated Convolutions for Efficient Multi-Scale Time to Contact Computation in Julia

Alexander Amini, Berthold K. P. Horn+1 more|arXiv (Cornell University)|Dec 28, 2016

Advanced Image and Video Retrieval Techniques7 references3 citations

TL;DR

This paper presents FastConv, an optimized n-dimensional convolution library in Julia for real-time multi-scale time-to-contact (TTC) estimation in machine vision. By accelerating convolutions through algorithmic optimization and memory efficiency, the method achieves over a 10x reduction in compute time and memory usage compared to existing approaches, enabling real-time TTC computation at ~10 Hz with multi-scale refinement and sub-millisecond per-frame performance on synthetic video data.

ABSTRACT

Convolutions have long been regarded as fundamental to applied mathematics, physics and engineering. Their mathematical elegance allows for common tasks such as numerical differentiation to be computed efficiently on large data sets. Efficient computation of convolutions is critical to artificial intelligence in real-time applications, like machine vision, where convolutions must be continuously and efficiently computed on tens to hundreds of kilobytes per second. In this paper, we explore how convolutions are used in fundamental machine vision applications. We present an accelerated n-dimensional convolution package in the high performance computing language, Julia, and demonstrate its efficacy in solving the time to contact problem for machine vision. Results are measured against synthetically generated videos and quantitatively assessed according to their mean squared error from the ground truth. We achieve over an order of magnitude decrease in compute time and allocated memory for comparable machine vision applications. All code is packaged and integrated into the official Julia Package Manager to be used in various other scenarios.

Motivation & Objective

• To develop a high-performance, memory-efficient convolution library tailored for real-time machine vision applications.
• To accelerate time-to-contact (TTC) and focus of expansion (FOE) computation in autonomous systems using n-dimensional convolutions.
• To enable real-time, low-latency TTC estimation on resource-constrained embedded devices via multi-scale resolution processing.
• To integrate the optimized convolution and TTC pipeline into the official Julia Package Manager for broad reproducibility and reuse.
• To evaluate performance gains using synthetic video sequences with ground-truth TTC for quantitative validation.

Proposed method

• Implementation of a fast, n-dimensional convolution algorithm (FastConv) in Julia, leveraging the language’s high-performance computing capabilities.
• Use of discrete derivatives via finite differences to compute image gradients $E_x$, $E_y$, and $E_t$ from video frames.
• Application of multi-scale down-sampling with progressive refinement to optimize TTC estimation accuracy.
• Employment of block averaging and low-pass filtering to reduce noise in high-resolution images before convolution.
• Integration of separable kernel detection to reduce computational complexity from $O(n^2)$ to $O(n)$ for common kernels like Gaussian and derivatives.
• Adoption of FFT-based convolution for large kernel sizes, extending applicability beyond direct convolution.

Experimental results

Research questions

• RQ1 Can a Julia-based convolution library achieve a 10x performance improvement over existing implementations for machine vision workloads?
• RQ2 How does multi-scale resolution processing affect the accuracy of time-to-contact estimation in synthetic video sequences?
• RQ3 To what extent can memory allocation be reduced while maintaining high-precision TTC and FOE computation?
• RQ4 Can the accelerated convolution method support real-time processing at 10–30 Hz on standard hardware?
• RQ5 How effective is progressive down-sampling in improving TTC estimation without prior environmental knowledge?

Key findings

• FastConv achieved over a 10x reduction in both compute time and allocated memory compared to baseline implementations.
• The multi-scale TTC algorithm outperformed static down-sampling methods, reducing mean squared error (MSE) by a significant margin on synthetic video data.
• Real-time performance was achieved at approximately 30 Hz for the base TTC algorithm and 10 Hz for the multi-scale variant on standard-definition video sequences.
• The implementation demonstrated sub-millisecond per-frame processing time, suitable for embedded and real-time robotic control systems.
• The integration of FastConv into the Julia Package Manager enables immediate reuse across diverse applications with minimal setup.
• Separable kernel detection reduced computational complexity from $O(n^2)$ to $O(n)$, further enhancing performance for common machine vision kernels.