[Paper Review] Visual Weather Temperature Prediction

Wei-Ta Chu, Kai-Chia Ho+1 more|arXiv (Cornell University)|Jan 25, 2018

Image Enhancement Techniques15 references3 citations

TL;DR

This paper proposes a deep learning approach for visual weather temperature prediction using only image data, employing a CNN for single-image temperature estimation and an RNN with LSTM for forecasting the temperature of the last image in a sequence. The method achieves state-of-the-art performance, particularly in temporal forecasting, and demonstrates that visual cues—especially from sky and foliage regions—significantly improve prediction accuracy.

ABSTRACT

In this paper, we attempt to employ convolutional recurrent neural networks for weather temperature estimation using only image data. We study ambient temperature estimation based on deep neural networks in two scenarios a) estimating temperature of a single outdoor image, and b) predicting temperature of the last image in an image sequence. In the first scenario, visual features are extracted by a convolutional neural network trained on a large-scale image dataset. We demonstrate that promising performance can be obtained, and analyze how volume of training data influences performance. In the second scenario, we consider the temporal evolution of visual appearance, and construct a recurrent neural network to predict the temperature of the last image in a given image sequence. We obtain better prediction accuracy compared to the state-of-the-art models. Further, we investigate how performance varies when information is extracted from different scene regions, and when images are captured in different daytime hours. Our approach further reinforces the idea of using only visual information for cost efficient weather prediction in the future.

Motivation & Objective

• To develop a deep learning model that estimates ambient temperature from a single outdoor image using only visual features.
• To improve temperature prediction accuracy by modeling temporal evolution in visual appearance across image sequences.
• To investigate how performance varies when extracting visual features from different scene regions (e.g., sky vs. ground).
• To analyze the impact of image capture time (daytime hours) on prediction performance.
• To identify which visual regions or cues are most informative for temperature estimation.

Proposed method

• A convolutional neural network (CNN) is used to extract visual features from a single outdoor image, followed by a regression layer to predict temperature.
• For sequential image input, a bidirectional Long Short-Term Memory (LSTM) network is employed to model temporal dependencies in visual appearance across frames.
• The model is trained end-to-end on a large-scale dataset of outdoor images captured by webcams over time.
• Visual saliency for temperature prediction is analyzed by computing color variation (standard deviation of RGB values) across time in non-overlapping image blocks.
• Performance is evaluated using RMSE and compared against state-of-the-art methods on real-world webcam datasets.
• The method evaluates temperature prediction accuracy when features are extracted only from sky regions or ground regions, and across different daytime hours (e.g., 11 am).

Experimental results

Research questions

• RQ1 Can a CNN-based model achieve high accuracy in estimating ambient temperature from a single outdoor image using only visual data?
• RQ2 Does incorporating temporal modeling via an RNN improve temperature prediction accuracy compared to single-image baselines?
• RQ3 How does the performance of temperature prediction vary when visual features are extracted from different regions of the scene (e.g., sky vs. ground)?
• RQ4 How does the time of day (e.g., morning vs. noon) affect the reliability of visual temperature estimation?
• RQ5 Which visual regions or cues (e.g., sky, foliage, building facades) are most informative for temperature prediction?

Key findings

• The proposed RNN-based model outperforms existing state-of-the-art methods in predicting the temperature of the last image in a sequence, demonstrating the benefit of modeling temporal evolution.
• Performance improves with larger training data volumes, confirming the scalability of deep learning for visual temperature estimation.
• Images captured around 11 am yield the most robust predictions, with the lowest average RMSE, due to optimal lighting and visual clarity.
• The sky region contributes significantly to temperature prediction, especially in clear-sky conditions, while foliage and building facades also show high saliency in certain scenes.
• Failure cases are most common when consecutive-day temperatures fluctuate sharply, indicating challenges in capturing rapid weather changes.
• Saliency maps reveal that regions with high color variation across days—such as tree leaves and parts of building facades—are more informative for temperature estimation.