[Paper Review] A Benchmark for Temporal Color Constancy

Yanlin Qian, Jani Käpylä+3 more|arXiv (Cornell University)|Mar 8, 2020

Color Science and Applications34 references4 citations

TL;DR

This paper introduces TCC-benchmark, the largest real-world temporal color constancy dataset with 600 high-resolution video sequences, and proposes TCC-Net, a novel 2D-LSTM-based deep learning model that leverages multi-frame temporal information for superior illumination estimation. TCC-Net achieves state-of-the-art performance, reducing mean angular error by 40% on TCC-benchmark and 30% on SFU Gray Ball compared to prior methods.

ABSTRACT

Temporal Color Constancy (CC) is a recently proposed approach that challenges the conventional single-frame color constancy. The conventional approach is to use a single frame - shot frame - to estimate the scene illumination color. In temporal CC, multiple frames from the view finder sequence are used to estimate the color. However, there are no realistic large scale temporal color constancy datasets for method evaluation. In this work, a new temporal CC benchmark is introduced. The benchmark comprises of (1) 600 real-world sequences recorded with a high-resolution mobile phone camera, (2) a fixed train-test split which ensures consistent evaluation, and (3) a baseline method which achieves high accuracy in the new benchmark and the dataset used in previous works. Results for more than 20 well-known color constancy methods including the recent state-of-the-arts are reported in our experiments.

Motivation & Objective

• To address the lack of large-scale, realistic temporal color constancy datasets for benchmarking.
• To evaluate the performance of over 20 single-frame and temporal color constancy methods under a fixed train-test split.
• To propose a strong, efficient baseline model (TCC-Net) that outperforms existing methods on both new and prior benchmarks.
• To enable fair, reproducible evaluation of temporal color constancy methods through open-sourced datasets and code.
• To explore architectural design choices such as 2D-LSTM, SqueezeNet backbone, and variable-length input sequences for optimal performance.

Proposed method

• The TCC-benchmark comprises 600 real-world video sequences captured with a high-resolution mobile phone, covering diverse indoor and outdoor scenes under varying weather and daylight conditions.
• A fixed train-test split ensures consistent evaluation across all methods, with sequences ranging from 3 to 17 frames in length.
• TCC-Net employs a two-branch architecture: one branch processes the shot frame using SqueezeNet for feature extraction, and another processes a pseudo-sequence of preceding frames using 2D-LSTM to model spatial-temporal dependencies.
• The 2D-LSTM module captures both spatial and temporal correlations by applying convolutional recurrence over 2D feature maps, with kernel size K=5 and hidden dimension H=128 for optimal trade-off.
• The model is trained end-to-end to predict the illuminant chromaticity, with loss minimized via mean angular error on the ground-truth labels.
• The framework supports variable-length input sequences, enabling deployment in real-world viewfinder scenarios where sequence length is unknown.

Experimental results

Research questions

• RQ1 How does the performance of temporal color constancy methods compare to single-frame methods on a realistic, large-scale benchmark?
• RQ2 What architectural components (e.g., 1D vs. 2D-LSTM, backbone network, sequence length) yield the best performance in temporal color constancy?
• RQ3 Can a lightweight, mobile-friendly model achieve state-of-the-art accuracy in temporal color constancy?
• RQ4 How does the proposed TCC-Net generalize across different datasets, including the previously used SFU Gray Ball dataset?
• RQ5 To what extent do spatial-temporal features learned by 2D-LSTM improve illumination estimation compared to single-frame or 1D-LSTM baselines?

Key findings

• TCC-Net achieves a mean angular error of 1.46° on the TCC-benchmark, representing a 40% improvement over the best single-frame method and a 27% improvement over the best prior temporal method.
• The 2D-LSTM variant of TCC-Net (Model G) achieves the best performance, outperforming 1D-LSTM and other configurations, with a mean angular error of 1.46° and a memory footprint of 68.8 MB.
• Using SqueezeNet as the backbone reduces memory footprint to 6.6 MB (Model C) while maintaining high accuracy, making it suitable for mobile deployment.
• The two-branch architecture with a pseudo-sequence branch improves performance by 12.7% in mean error compared to a single-branch design.
• The optimal kernel size for 2D-LSTM is K=5, and the best hidden dimension is H=128, balancing underfitting and overfitting.
• TCC-Net generalizes well, achieving a 33% lower mean angular error than the best single-frame method and a 30% lower error than the best temporal method on the SFU Gray Ball dataset.