[Paper Review] Sunrise or Sunset: Selective Comparison Learning for Subtle Attribute Recognition

Hong-Yu Zhou, Bin-Bin Gao+1 more|arXiv (Cornell University)|Jul 20, 2017

Advanced Image and Video Retrieval Techniques23 references5 citations

TL;DR

This paper proposes SoSNet, a selective comparison learning framework that improves recognition of subtle visual attributes like sunrise vs. sunset by training on discriminative image pairs. The method achieves state-of-the-art performance on both the novel Sunrise or Sunset (SoS) dataset and temperature estimation, outperforming human baselines and existing deep learning models by significant margins.

ABSTRACT

The difficulty of image recognition has gradually increased from general category recognition to fine-grained recognition and to the recognition of some subtle attributes such as temperature and geolocation. In this paper, we try to focus on the classification between sunrise and sunset and hope to give a hint about how to tell the difference in subtle attributes. Sunrise vs. sunset is a difficult recognition task, which is challenging even for humans. Towards understanding this new problem, we first collect a new dataset made up of over one hundred webcams from different places. Since existing algorithmic methods have poor accuracy, we propose a new pairwise learning strategy to learn features from selective pairs of images. Experiments show that our approach surpasses baseline methods by a large margin and achieves better results even compared with humans. We also apply our approach to existing subtle attribute recognition problems, such as temperature estimation, and achieve state-of-the-art results.

Motivation & Objective

• To address the challenge of recognizing subtle visual attributes that are difficult even for humans, such as distinguishing sunrise from sunset in real-world photos.
• To build a new benchmark dataset, SoS, comprising over 100 webcams capturing sunrise and sunset scenes for fine-grained attribute recognition.
• To develop a novel pairwise learning framework that selects informative image pairs to improve feature learning for subtle attribute classification.
• To demonstrate the generalizability of the method by applying it to other subtle attribute tasks, such as estimating ambient temperature from images.

Proposed method

• Proposes a selective comparison learning strategy that trains a deep neural network on curated pairs of images, focusing on the most discriminative samples for improved feature learning.
• Uses a Siamese-like network architecture with a contrastive loss to learn embedding representations that emphasize differences between image pairs.
• Applies a sampling strategy that prioritizes image pairs with high uncertainty or low confidence predictions to improve model generalization.
• Adapts the framework for regression tasks by replacing softmax loss with mean squared error, enabling temperature estimation from images.
• Trains an independent SoSNet per scene on the Glasner dataset and extracts Pool4 features for linear ν-SVR regression.
• Employs a learning rate schedule and hyperparameter settings consistent with prior work for fair comparison.

Experimental results

Research questions

• RQ1 Can a deep learning model outperform humans in distinguishing subtle visual attributes like sunrise and sunset in real-world images?
• RQ2 How effective is selective comparison learning using curated image pairs in improving recognition accuracy for fine-grained, subtle attribute classification?
• RQ3 Can the proposed SoSNet framework generalize beyond the sunrise/sunset task to other subtle attribute recognition problems, such as temperature estimation?
• RQ4 What is the impact of different loss functions (e.g., contrastive vs. mean squared error) on performance in both classification and regression settings?

Key findings

• The proposed SoSNet achieves 73.0% accuracy on the SoS dataset, significantly outperforming human volunteers (64.0%) and baseline models (51.7–56.3%).
• On the temperature estimation benchmark, SoSNet achieves the highest R² (0.90) and lowest RMSE (3.98) on scene (j), outperforming all prior methods including CNN and Transient Attributes.
• SoSNet achieves state-of-the-art results across all 10 scenes in the Glasner temperature dataset, with R² values ranging from 0.73 to 0.90 and RMSE from 2.53 to 6.26.
• The model's performance surpasses traditional CNNs and other SOTA methods, indicating that selective pairwise learning is highly effective for subtle attribute recognition.
• The ablation shows that the selective sampling of image pairs significantly improves learning efficiency and accuracy compared to random or non-selective pairing.
• The method generalizes well to other subtle attribute tasks, demonstrating robustness and transferability beyond the original sunrise/sunset classification task.