[Paper Review] Answering Image Riddles using Vision and Reasoning through Probabilistic Soft Logic

Somak Aditya, Yezhou Yang+2 more|arXiv (Cornell University)|Nov 17, 2016

Advanced Image and Video Retrieval Techniques21 references3 citations

TL;DR

This paper introduces a novel task of answering image riddles—puzzles requiring both visual understanding and commonsense reasoning—using a Probabilistic Soft Logic (PSL) framework. It proposes a method that combines object detection from deep learning with ConceptNet-based knowledge reasoning to infer abstract, shared concepts across multiple images, achieving strong performance on a newly curated dataset of 3,000 riddles with human-validated annotations.

ABSTRACT

In this work, we explore a genre of puzzles ("image riddles") which involves a set of images and a question. Answering these puzzles require both capabilities involving visual detection (including object, activity recognition) and, knowledge-based or commonsense reasoning. We compile a dataset of over 3k riddles where each riddle consists of 4 images and a groundtruth answer. The annotations are validated using crowd-sourced evaluation. We also define an automatic evaluation metric to track future progress. Our task bears similarity with the commonly known IQ tasks such as analogy solving, sequence filling that are often used to test intelligence. We develop a Probabilistic Reasoning-based approach that utilizes probabilistic commonsense knowledge to answer these riddles with a reasonable accuracy. We demonstrate the results of our approach using both automatic and human evaluations. Our approach achieves some promising results for these riddles and provides a strong baseline for future attempts. We make the entire dataset and related materials publicly available to the community in ImageRiddle Website (http://bit.ly/22f9Ala).

Motivation & Objective

• To introduce a new benchmark task—answering image riddles—that tests both visual perception and high-level reasoning.
• To collect and validate a large-scale dataset of 3,000 image riddles with four images and a single word answer, ensuring semantic coherence and difficulty through crowd-sourcing.
• To develop a reasoning system that infers abstract, shared concepts from visual detections using probabilistic commonsense knowledge.
• To establish a strong baseline for future research in visual reasoning by combining vision models with structured knowledge reasoning.
• To provide an automatic evaluation metric and human evaluation protocol to track progress in this new task.

Proposed method

• Use state-of-the-art image classification models (e.g., Clarifai, ResNet) to extract top-5 class-label predictions for each image in a riddle.
• Map detected labels to a broader semantic space using ConceptNet 5, a knowledge graph encoding relations between words and phrases.
• Apply Probabilistic Soft Logic (PSL) to perform structured reasoning over the detected labels and ConceptNet knowledge, inferring the most probable shared concept across all images.
• Implement a multi-stage pipeline: (1) visual detection, (2) concept expansion via ConceptNet, (3) probabilistic reasoning using PSL to rank candidate answers.
• Incorporate bias correction via a GUR (Global Unbiased Ranking) module to improve answer quality and reduce overconfidence in predictions.
• Optimize hyperparameters (e.g., similarity thresholds, confidence thresholds) using random search on a held-out set of 500 riddles.

Experimental results

Research questions

• RQ1 Can a system combining visual detection and commonsense reasoning effectively solve image riddles that require abstract conceptual connections?
• RQ2 How does integrating structured knowledge (via ConceptNet) improve answer accuracy compared to vision-only baselines?
• RQ3 To what extent does probabilistic reasoning with PSL enhance the model’s ability to infer non-obvious, shared concepts across diverse images?
• RQ4 How do human evaluators perceive the correctness and intelligence of system-generated answers in comparison to vision-only models?
• RQ5 Can a unified framework generalize across riddles with unique, non-redundant answers and diverse visual cues?

Key findings

• The GUR+All variant of the proposed method achieved the highest word2vec-based accuracy on the test set, outperforming vision-only baselines.
• Human evaluation on Amazon Mechanical Turk showed that the GUR+All system received significantly higher intelligence ratings (2.2/4) compared to Clarifai (2.0/4) and ResNet (1.8/4).
• The system demonstrated improved correctness and explainability, with a higher proportion of answers falling into the highest correctness and intelligence bins in human evaluation.
• The accuracy jump after the reasoning stage (RR) in the pipeline indicates that PSL-based reasoning is the primary driver of performance gains.
• The dataset of 3,000 riddles was validated via crowd-sourcing, confirming its semantic coherence and appropriate difficulty level.
• The proposed method generalizes well, as it infers concepts from a large vocabulary (ConceptNet’s ~200k entries) based on a small set of detected labels, enabling discovery of abstract, non-trivial connections.