[Paper Review] Image similarity using Deep CNN and Curriculum Learning

Srikar Appalaraju, Vineet Chaoji|arXiv (Cornell University)|Sep 26, 2017

Advanced Image and Video Retrieval Techniques29 references61 citations

TL;DR

The paper introduces SimNet, a deep siamese network for image similarity that uses online pair mining inspired by curriculum learning and a multi-scale embedding to better capture fine-grained similarities than traditional CNNs.

ABSTRACT

Image similarity involves fetching similar looking images given a reference image. Our solution called SimNet, is a deep siamese network which is trained on pairs of positive and negative images using a novel online pair mining strategy inspired by Curriculum learning. We also created a multi-scale CNN, where the final image embedding is a joint representation of top as well as lower layer embedding's. We go on to show that this multi-scale siamese network is better at capturing fine grained image similarities than traditional CNN's.

Motivation & Objective

• Motivate the task of finding visually similar images given a reference image.
• Develop a deep learning solution (SimNet) using a siamese architecture for positive/negative pair training.
• Introduce an online pair mining strategy inspired by curriculum learning to select informative training pairs.
• Incorporate a multi-scale CNN to fuse top and lower layer embeddings for richer image representations.

Proposed method

• Build a deep siamese network (SimNet) trained on positive/negative image pairs.
• Apply an online pair mining strategy inspired by Curriculum Learning to select informative pairs during training.
• Design a multi-scale CNN where the final embedding is a joint representation of top and lower layer embeddings.
• Compare the multi-scale siamese embedding against traditional CNN-based embeddings for image similarity tasks.

Experimental results

Research questions

• RQ1 How effective is SimNet at retrieving visually similar images using a siamese architecture?
• RQ2 Does online pair mining guided by Curriculum Learning improve training efficiency and model performance?
• RQ3 Does combining top and lower layer embeddings in a multi-scale CNN yield better fine-grained similarity than standard CNN embeddings?

Key findings

• The multi-scale siamese network better captures fine-grained image similarities than traditional CNNs.
• Online pair mining guided by Curriculum Learning is used to train the siamese network on informative image pairs.
• The proposed approach emphasizes embedding fusion from multiple CNN layers to improve similarity judgments.