[Paper Review] Detection and Description of Change in Visual Streams

Davis Gilton, Ruotian Luo+2 more|arXiv (Cornell University)|Mar 27, 2020

Multimodal Machine Learning Applications53 references4 citations

TL;DR

This paper proposes a semi-supervised framework that leverages natural language descriptions of change to improve visual stream change detection, using a regularized graph cut based detector that integrates learned representations for change evidence and consistency. The method significantly outperforms image-only baselines, especially with limited labeled data, demonstrating that language supervision enhances detection accuracy and generalization in visual change analysis.

ABSTRACT

This paper presents a framework for the analysis of changes in visual streams: ordered sequences of images, possibly separated by significant time gaps. We propose a new approach to incorporating unlabeled data into training to generate natural language descriptions of change. We also develop a framework for estimating the time of change in visual stream. We use learned representations for change evidence and consistency of perceived change, and combine these in a regularized graph cut based change detector. Experimental evaluation on visual stream datasets, which we release as part of our contribution, shows that representation learning driven by natural language descriptions significantly improves change detection accuracy, compared to methods that do not rely on language.

Motivation & Objective

• To address the challenge of detecting and describing relevant changes in visual streams, such as satellite or street-view imagery, while ignoring nuisance changes like lighting or perspective shifts.
• To develop a joint learning framework that improves change detection accuracy by incorporating natural language descriptions of changes, even with limited labeled data.
• To create and release two new datasets—CLEVR-Sequence and Street Change—for visual stream change detection and captioning, supporting semi-supervised learning.
• To demonstrate that language-informed representations enhance change detection beyond what is possible with image-only supervision.

Proposed method

• A semi-supervised training regimen is proposed that jointly trains a change caption generator and a change discriminator using both labeled image pairs with descriptions and large amounts of unlabeled image pairs.
• The method uses a regularized graph cut cost function that incorporates two components: (1) change evidence from learned image pair representations, and (2) consistency of perceived change across image pairs straddling a changepoint.
• The consistency loss is computed using the hidden state of the caption generator and the output of the change discriminator, ensuring that similar changes are consistently detected across different image pairs.
• The framework employs a Siamese-like network architecture for image pair encoding, with the final hidden representation used to compute change scores.
• A hyperparameter λ controls the trade-off between change evidence and consistency, optimized on a held-out validation set.
• The approach is evaluated on two new datasets: CLEVR-Sequence (synthetic) and Street Change (real-world), with performance measured via precision-recall and mean average precision (mAP).

Experimental results

Research questions

• RQ1 Can natural language descriptions of changes in visual streams improve the accuracy of change detection compared to image-only methods?
• RQ2 To what extent does semi-supervised learning using unlabeled image pairs enhance performance when labeled data is scarce?
• RQ3 How does incorporating representation consistency across image pairs affect the robustness and accuracy of change detection?
• RQ4 Does joint learning of captioning and change detection lead to better generalization than training each task in isolation?

Key findings

• The proposed method with language-informed representations (RC) achieves 87.5% mAP on CLEVR-Sequence and 73.0% mAP on Street Change, significantly outperforming image-only baselines.
• Adding unlabeled data consistently improves performance, with the semi-supervised approach surpassing fully supervised baselines when labeled data is limited (e.g., L=1k).
• The regularized graph cut with language supervision (RC) achieves 92.9% precision at recall 1.0 on CLEVR-Sequence with a 4-frame tolerance window, compared to 65.8% for the image-only baseline (Step-IO).
• The representation consistency loss alone (RC-IO λ=0) improves detection over stepwise methods, showing that consistency is a valuable inductive bias for change detection.
• On Street Change, the method achieves 73.2% mAP with a 4-frame tolerance, demonstrating strong generalization to real-world visual streams.
• The performance gain from language supervision is most pronounced when labeled data is scarce, confirming that language provides a richer form of supervision than change timestamps alone.